diff --git a/.github/scripts/otel-naming/check_otel_naming.py b/.github/scripts/otel-naming/check_otel_naming.py index 68da7a34e4..e45216ecf9 100644 --- a/.github/scripts/otel-naming/check_otel_naming.py +++ b/.github/scripts/otel-naming/check_otel_naming.py @@ -49,7 +49,9 @@ Layers L3 tempo : docker/telemetry/tempo.yaml (span filter tags) L4 dashboards: docker/telemetry/grafana/dashboards/*.json (PromQL labels; Loki/LogQL queries exempt -- see LOG_QUERY_DATASOURCES) - L5 runbook : docs/telemetry-runbook.md (attr tables) + L5 docs : every *.md under docs/ and docker/telemetry/ (attr tables; + the set is DISCOVERED, never enumerated -- see + RULE_E_DOC_ROOTS) L6 metrics : MetricsRegistry.cpp instrument labels (native-metric label keys, a valid dashboard-label source besides L1) L1-metrics : src/**/*MetricNames.h, include/**/*MetricNames.h (ground truth @@ -77,11 +79,17 @@ Rules (each FAILS the build, when its inputs are present) captures or an in-query `| regexp` stage, so they have no L1/L6 source to resolve against (see LOG_QUERY_DATASOURCES). The exemption is per QUERY, not per file, so a mixed dashboard still has its PromQL panels checked. - E No dotted `xrpl..` attribute key in the runbook (only the - L1 resource attrs xrpl.network.* and the EXTERNAL_INFRA_LABELS dotted - form -- xrpl.work.item/.branch/.node.role -- may be dotted). Span names, - filenames, - OTel-standard keys, and metric labels are not flagged. + E No dotted `xrpl..` attribute key in any L5 doc. The doc set + is DISCOVERED, not listed: every `*.md` under RULE_E_DOC_ROOTS, so a doc + added or renamed inside a root is covered without editing this script. + Only the L1 resource + attrs xrpl.network.* and the EXTERNAL_INFRA_LABELS dotted form -- + xrpl.work.item/.branch/.node.role -- may be dotted. Span names, filenames, + OTel-standard keys, and metric labels are not flagged. A doc that names the + wrong form as a deliberate counter-example opts that mention out with a + key-scoped marker (RULE_E_MARKER_SYNTAX), which exempts ONLY the keys it + lists on the line it appears on -- any other dotted token on that same line + still fails. I No string literals as METRIC instrument names or label keys. The mirror of Rule F for metrics: the name passed to an XRPL_METRIC_* macro or to a meter->Create* factory, and the label KEYS in its label set, must reference @@ -108,6 +116,13 @@ Warnings (printed, but do NOT fail the build) *SpanNames.h (single source of truth); defining one in-place bypasses the naming rules. A warning (not a failure) because the argument may instead be a legitimately dynamic local (e.g. a computed span-name leaf). + E A RULE_E_REQUIRED_DOCS anchor is not present in the tree (renamed, moved, + or deleted -- so doc discovery can no longer be trusted to have found the + docs that matter), an allow-dotted marker names no key (so it exempts + nothing), or it names a key the line no longer mentions (a stale + exemption). Warnings, not failures, because presence-gating must keep + working on partial branches -- but the skip is now visible instead of + silent. L A literal metric name in a family that has no *MetricNames.h constants yet. Rule I's ratchet defers these rather than failing the build on the whole pre-existing metric surface at once; the warning is what keeps the @@ -520,7 +535,7 @@ def main() -> None: run_rule_b_collector(root, l1_keys, report) run_rule_c_tempo(root, l1_keys, report) run_rule_d_dashboards(root, l1_keys, metric_labels, report) - run_rule_e_runbook(root, l1_keys, report) + run_rule_e_docs(root, l1_keys, report) # --- Metric rules I/J/K -------------------------------------------------- # The metric-name counterpart of the span rules above. Rule I is the mirror @@ -1038,6 +1053,107 @@ EXTERNAL_INFRA_LABELS = { } +# L5 doc layer (Rule E): the doc trees whose markdown publishes span-attribute +# keys a reader is expected to copy into a TraceQL/PromQL query. A dotted +# `xrpl.*` key left in any of them hands out a query that silently matches +# nothing, so the whole tree is checked, not just the operator runbook: +# * docs/ operator runbook, telemetry glossary, and the +# build/telemetry guides (recursively) +# * docker/telemetry/ stack bring-up guide; its TraceQL/PromQL examples +# are meant to be pasted verbatim +# +# The set is DISCOVERED (every `*.md` under these roots), never enumerated. Two +# reasons this beats a hardcoded path list: +# * it cannot go stale -- a telemetry doc added, renamed, or moved within a +# root is picked up with no edit here, which is the same "derive it, do not +# hardcode it" principle the rest of this script follows (design +# principle 1), and coverage therefore grows with the docs; and +# * it names only doc ROOTS that ship on the default branch, so the rule can +# never be wired to a path that does not exist there. +# Each root is presence-gated (design principle 2): a branch that carries only +# one of them still gets that one checked. Both roots are also CI path-triggers +# for this check -- see `.github/workflows/on-pr.yml`. +RULE_E_DOC_ROOTS = ( + Path("docs"), + Path("docker") / "telemetry", +) + +# The anchor doc(s) discovery is REQUIRED to find. Discovery on its own can pass +# vacuously: if a root were renamed away, or emptied of markdown, Rule E would +# report a clean zero/near-zero-file run and nobody would notice the layer had +# stopped being checked. Demanding the one doc that is the whole reason Rule E +# exists turns that silent green into a visible warning (plus a count on the OK +# line). Deliberately tiny: this is a tripwire, not a second doc list. Every +# entry must live under RULE_E_DOC_ROOTS, or discovery could never find it. +RULE_E_REQUIRED_DOCS = (Path("docs") / "telemetry-runbook.md",) + + +def rule_e_docs(root: Path) -> List[Path]: + """Discover the Rule E doc set: every `*.md` under RULE_E_DOC_ROOTS. + + Returns repo-relative paths, sorted for stable reporting and de-duplicated + so overlapping roots (say `docs` and `docs/build`) cannot check one file + twice. A root absent from the tree is skipped rather than treated as an + error, so presence gating applies to the roots as it does to every other + layer.""" + found: Set[Path] = set() + for rel_root in RULE_E_DOC_ROOTS: + base = root / rel_root + if not base.is_dir(): + continue + found.update(p.relative_to(root) for p in base.rglob("*.md") if p.is_file()) + return sorted(found) + + +# Line-scoped AND key-scoped opt-out for Rule E. Put it on a line that names a +# dotted key as a deliberate counter-example ("`tx_hash`, not `xrpl.tx.hash`") — +# a mention, not a published attribute key — and list exactly the keys that line +# is allowed to mention: +# +# ... use `tx_hash`, not `xrpl.tx.hash`. +# +# +# The keys are part of the marker so an exemption cannot widen silently: a dotted +# token added to an already-marked line later, and not named in the marker, still +# fails. A marker with an empty key list therefore exempts NOTHING (and warns) — +# it is not a blanket line opt-out. NEVER use the marker to keep a real attribute +# table dotted; fix the table instead. +RULE_E_MARKER_SYNTAX = "" +# The marker itself; group(1) is the raw key list (empty for the bare form). +# `[^>]` keeps the match inside one comment, so a marker cannot swallow the rest +# of the line, and the whole thing is matched per line (never across lines). +RULE_E_MARKER = re.compile(r"") +# Keys inside the marker are separated by commas and/or whitespace. +RULE_E_MARKER_KEY_SEP = re.compile(r"[,\s]+") +# The only doc form Rule E flags: a backticked dotted `xrpl..`. +RULE_E_DOTTED_TOKEN = re.compile(r"`(xrpl\.[a-z][a-z0-9_.]*)`") + + +def rule_e_allowed_keys(line: str) -> Tuple[Set[str], int]: + """Parse every Rule-E allow-dotted marker on one line. + + Returns `(keys named by the markers, number of markers seen)`. The count is + returned separately so the caller can tell "no marker" (enforce everything, + silently) from "a marker that names no key" (enforce everything, and warn + that the marker does nothing). + + Keys may be written bare or backticked and separated by commas and/or + spaces, so both of these parse to the same two keys:: + + + + """ + keys: Set[str] = set() + count = 0 + for m in RULE_E_MARKER.finditer(line): + count += 1 + for raw in RULE_E_MARKER_KEY_SEP.split(m.group(1)): + token = raw.strip().strip("`") + if token: + keys.add(token) + return keys, count + + # Datasource types whose query language draws its label names from the log # stream at query time rather than from anything this repo's OTel code emits. # @@ -1228,20 +1344,33 @@ def run_rule_d_dashboards( report.ok(note + ")") -def run_rule_e_runbook(root: Path, l1_keys: Set[str], report: Report) -> None: - path = root / "docs" / "telemetry-runbook.md" - if not path.is_file(): - report.skip("E", "runbook not present") +def run_rule_e_docs(root: Path, l1_keys: Set[str], report: Report) -> None: + discovered = rule_e_docs(root) + if not discovered: + roots = ", ".join(str(rel) for rel in RULE_E_DOC_ROOTS) + report.skip("E", f"no doc-layer file present (no *.md under {roots})") return + # Presence-gating (design principle 2) must stay VISIBLE. Discovery cannot go + # stale, but it can go QUIET: a root renamed away, or stripped of its + # telemetry docs, still yields a clean run over whatever markdown is left, so + # the layer would stop being checked while the rule reported green. Warn per + # missing anchor — non-fatal, so a partial branch still passes — and name the + # count on the OK line, so the doc set cannot quietly shrink. + missing_required = [ + rel for rel in RULE_E_REQUIRED_DOCS if not (root / rel).is_file() + ] + for rel in missing_required: + report.warning( + "E", str(rel), "absent", "required Rule E doc not in tree (renamed?)" + ) if not l1_keys: report.skip("E", "no L1 key set to validate against") return - text = read_source(path) found = False # Only the dotted `xrpl..` attribute form is a violation. The # `xrpl.`-with-trailing-dot anchor is the discriminator: it matches the old # dotted attribute convention being migrated away from, while everything - # else legitimately dotted in the runbook does NOT match it — + # else legitimately dotted in these docs does NOT match it — # * span names (`consensus.round`, `tx.process`) no `xrpl.` prefix # * filenames (`xrpld.cfg`, `RCLConsensus.cpp`) `xrpld.`/`.cpp`, not `xrpl.` # * OTel-standard (`service.name`, `http.method`) no `xrpl.` prefix @@ -1254,19 +1383,57 @@ def run_rule_e_runbook(root: Path, l1_keys: Set[str], report: Report) -> None: # identities dotted (xrpl.work.item/.branch/.node.role -- see the alloy # pipeline that owns them), so also skip a token whose dotted-to-underscore # form is in that set. + # A doc that TEACHES the convention has to be able to name the wrong form as + # a counter-example ("`tx_hash`, not `xrpl.tx.hash`"). The allow-dotted + # marker is the opt-out for exactly that: the token is a mention, not a + # published attribute key. It follows the repo's existing inline-marker + # precedent (``), and is scoped BOTH to the line + # and to the keys it names, so it can neither exempt a whole table nor grow + # to cover a violation appended to an already-marked line. external_infra_dotted = {lbl.replace("_", ".") for lbl in EXTERNAL_INFRA_LABELS} - for m in re.finditer(r"`(xrpl\.[a-z][a-z0-9_.]*)`", text): - token = m.group(1) - if token in l1_keys: # legitimate dotted resource attr (xrpl.network.*) - continue - if token in external_infra_dotted: # perf-iac resource-attribute layer - continue - found = True - report.violation( - "E", str(path.relative_to(root)), token, "underscore, not dotted" - ) + for rel in discovered: + for lineno, line in enumerate(read_source(root / rel).splitlines(), start=1): + allowed, markers = rule_e_allowed_keys(line) + if markers and not allowed: + report.warning( + "E", + f"{rel}:{lineno}", + "allow-dotted", + f"marker names no key: exempts nothing. Use {RULE_E_MARKER_SYNTAX}", + ) + tokens = [m.group(1) for m in RULE_E_DOTTED_TOKEN.finditer(line)] + for token in tokens: + if token in l1_keys: # legitimate dotted resource attr (xrpl.network.*) + continue + if token in external_infra_dotted: # perf-iac resource-attr layer + continue + if token in allowed: # named counter-example on this line + continue + found = True + report.violation( + "E", + f"{rel}:{lineno}", + token, + "underscore, not dotted", + ) + # A key the marker names but the line no longer mentions is a stale + # exemption: harmless today, but it is how a marker silently starts + # covering more than the author reviewed. Flagged, never fatal. + for stale in sorted(allowed.difference(tokens)): + report.warning( + "E", f"{rel}:{lineno}", stale, "allow-dotted key not on this line" + ) if not found: - report.ok("E: runbook attribute references consistent with L1") + note = ( + "E: doc attribute references consistent with L1 " + f"({len(discovered)} file(s) checked" + ) + if missing_required: + note += ( + f", {len(missing_required)} of {len(RULE_E_REQUIRED_DOCS)} " + "required doc(s) absent" + ) + report.ok(note + ")") # --------------------------------------------------------------------------- diff --git a/.github/scripts/otel-naming/test_check_otel_naming.py b/.github/scripts/otel-naming/test_check_otel_naming.py index a206a0ce28..7942bcba1e 100644 --- a/.github/scripts/otel-naming/test_check_otel_naming.py +++ b/.github/scripts/otel-naming/test_check_otel_naming.py @@ -13,7 +13,7 @@ Run from anywhere: Each rule is exercised in isolation against a synthetic tree / synthetic L1 key set, covering positive (must flag), negative (must not flag), and boundary -cases. Rule E (runbook dotted-attribute detection) has the densest coverage +cases. Rule E (doc-layer dotted-attribute detection) has the densest coverage because its discriminator — the `xrpl..` prefix vs span names, filenames, OTel-standard keys, and metric labels — is the subtlest. """ @@ -23,6 +23,7 @@ import importlib.util import io import json import shutil +import subprocess import tempfile import unittest from pathlib import Path @@ -51,13 +52,25 @@ L1 = { def _run_rule_e(runbook_text: str): """Run Rule E against a synthetic runbook; return the flagged tokens.""" + return _run_rule_e_docs({"docs/telemetry-runbook.md": runbook_text})[0] + + +def _run_rule_e_docs(docs, l1_keys=None): + """Run Rule E against a synthetic doc set. + + `docs` maps a repo-relative path to that file's text; only the listed files + are created, so per-file presence gating can be exercised. Returns + (sorted flagged tokens, the Report) so location and skip/ok lines can be + asserted as well as the tokens.""" d = Path(tempfile.mkdtemp()) try: - (d / "docs").mkdir() - (d / "docs" / "telemetry-runbook.md").write_text(runbook_text) + for rel, text in docs.items(): + path = d / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) report = chk.Report() - chk.run_rule_e_runbook(d, set(L1), report) - return sorted(v[2] for v in report.violations) + chk.run_rule_e_docs(d, set(L1) if l1_keys is None else set(l1_keys), report) + return sorted(v[2] for v in report.violations), report finally: shutil.rmtree(d) @@ -170,7 +183,7 @@ class RuleERunbook(unittest.TestCase): d = Path(tempfile.mkdtemp()) try: report = chk.Report() - chk.run_rule_e_runbook(d, set(L1), report) + chk.run_rule_e_docs(d, set(L1), report) self.assertEqual(report.violations, []) self.assertTrue(any("SKIP: E" in s for s in report.skips)) finally: @@ -182,13 +195,593 @@ class RuleERunbook(unittest.TestCase): (d / "docs").mkdir() (d / "docs" / "telemetry-runbook.md").write_text("`xrpl.tx.hash`") report = chk.Report() - chk.run_rule_e_runbook(d, set(), report) + chk.run_rule_e_docs(d, set(), report) self.assertEqual(report.violations, []) self.assertTrue(any("SKIP: E" in s for s in report.skips)) finally: shutil.rmtree(d) +# The doc paths the Rule E tests build synthetic trees from. All four are real, +# committable files that live under RULE_E_DOC_ROOTS, so a test tree has the same +# shape discovery meets in the repo. GLOSSARY and BUILD_GUIDE additionally prove +# discovery is not runbook-shaped: BUILD_GUIDE sits one directory deeper, so a +# non-recursive glob would miss it. +RUNBOOK_DOC = "docs/telemetry-runbook.md" +GLOSSARY_DOC = "docs/telemetry-glossary.md" +BUILD_GUIDE_DOC = "docs/build/telemetry.md" +TESTING_DOC = "docker/telemetry/TESTING.md" +ALL_DOCS = (RUNBOOK_DOC, GLOSSARY_DOC, BUILD_GUIDE_DOC, TESTING_DOC) + +# Top-level trees that ship to the default branch, so anything the check is +# configured to read must live inside one of them. A configured path outside +# these would name a tree that is not part of the shipped repository: the rule +# would warn forever about a doc that can never appear, and the matching CI +# path-trigger would be dead weight. +COMMITTABLE_ROOTS = (Path("docs"), Path("docker")) + + +def _inside_any(rel: Path, roots) -> bool: + """True if `rel` is one of `roots` or lives underneath one of them.""" + return any(root == rel or root in rel.parents for root in roots) + + +def _git_tracks(root: Path, rel: Path) -> bool: + """True if git tracks `rel` (a file) or anything under it (a directory). + + Evidence that a configured path is genuinely part of the repository rather + than a local scratch or ignored directory.""" + out = subprocess.run( + ["git", "ls-files", "--", str(rel)], + cwd=str(root), + capture_output=True, + text=True, + ) + return bool(out.stdout.strip()) + + +class RuleEDocSet(unittest.TestCase): + """Rule E scans every discovered doc, not just the runbook. The glossary, the + build/telemetry guide and the stack testing guide all publish attribute keys + readers copy queries from, so a dotted key left in any of them is as + operator-breaking as one in the runbook.""" + + RUNBOOK = RUNBOOK_DOC + GLOSSARY = GLOSSARY_DOC + BUILD_GUIDE = BUILD_GUIDE_DOC + TESTING = TESTING_DOC + + def test_flags_dotted_attr_in_testing_guide(self): + # TESTING.md publishes copy-paste TraceQL/PromQL, so a dotted key there + # hands the reader a query that silently matches nothing. + tokens, report = _run_rule_e_docs( + {self.TESTING: '{name="tx.process" && span.`xrpl.tx.hash`="AB"}'} + ) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + self.assertEqual(report.violations[0][1], f"{self.TESTING}:1") + self.assertEqual(report.violations[0][3], "underscore, not dotted") + + def test_testing_guide_checked_when_every_other_doc_absent(self): + # Per-file presence gating: TESTING.md alone must not skip the rule. + tokens, report = _run_rule_e_docs({self.TESTING: "`xrpl.peer.version`"}) + self.assertEqual(tokens, ["xrpl.peer.version"]) + self.assertEqual(report.skips, []) + self.assertEqual(report.violations[0][1], f"{self.TESTING}:1") + # A violating run emits the violation, not an "ok" line. + self.assertEqual(report.checked, []) + + def test_testing_guide_alone_reports_one_clean_file(self): + tokens, report = _run_rule_e_docs({self.TESTING: "`tx_hash`"}) + self.assertEqual(tokens, []) + self.assertTrue(any("E:" in c and "1 file(s)" in c for c in report.checked)) + + def test_clean_testing_guide_passes_with_bare_keys(self): + # The real docker/telemetry/TESTING.md shape: bare/underscore attr keys + # plus dotted SPAN names, which must not be flagged. + tokens, report = _run_rule_e_docs( + {self.TESTING: '{name="tx.process" && span.tx_hash!=""} `ledger_seq`'} + ) + self.assertEqual(tokens, []) + self.assertEqual(report.violations, []) + + def test_exemptions_apply_in_testing_guide_too(self): + tokens, _ = _run_rule_e_docs( + {self.TESTING: "`xrpl.network.type` `xrpl.work.item`"} + ) + self.assertEqual(tokens, []) + + def test_flags_dotted_attr_in_glossary(self): + tokens, report = _run_rule_e_docs({self.GLOSSARY: "| `xrpl.tx.hash` | h |"}) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + self.assertEqual(report.violations[0][1], f"{self.GLOSSARY}:1") + + def test_flags_dotted_attr_in_nested_build_guide(self): + # One directory below the root: proves discovery recurses rather than + # globbing only the root's own children. + tokens, report = _run_rule_e_docs({self.BUILD_GUIDE: "`xrpl.consensus.mode`"}) + self.assertEqual(tokens, ["xrpl.consensus.mode"]) + self.assertEqual(report.violations[0][1], f"{self.BUILD_GUIDE}:1") + + def test_location_carries_the_offending_line_number(self): + tokens, report = _run_rule_e_docs( + {self.GLOSSARY: "clean\nstill clean\n| `xrpl.tx.hash` |"} + ) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + self.assertEqual(report.violations[0][1], f"{self.GLOSSARY}:3") + + def test_flags_across_all_four_docs(self): + tokens, report = _run_rule_e_docs( + { + self.RUNBOOK: "`xrpl.tx.hash`", + self.GLOSSARY: "`xrpl.peer.id`", + self.BUILD_GUIDE: "`xrpl.ledger.seq`", + self.TESTING: "`xrpl.rpc.command`", + } + ) + self.assertEqual( + tokens, + [ + "xrpl.ledger.seq", + "xrpl.peer.id", + "xrpl.rpc.command", + "xrpl.tx.hash", + ], + ) + self.assertEqual( + sorted(v[1] for v in report.violations), + sorted( + [ + f"{self.RUNBOOK}:1", + f"{self.GLOSSARY}:1", + f"{self.BUILD_GUIDE}:1", + f"{self.TESTING}:1", + ] + ), + ) + + def test_other_doc_checked_when_runbook_absent(self): + # Presence gating is per discovered file: a branch without the runbook + # still gets the docs it does carry checked, instead of the rule skipping. + tokens, report = _run_rule_e_docs({self.GLOSSARY: "`xrpl.tx.hash`"}) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + self.assertEqual(report.skips, []) + + def test_absent_doc_is_not_an_error(self): + tokens, report = _run_rule_e_docs({self.RUNBOOK: "`tx_hash`"}) + self.assertEqual(tokens, []) + self.assertTrue(any("1 file(s)" in c for c in report.checked)) + + def test_clean_doc_set_reports_file_count(self): + tokens, report = _run_rule_e_docs( + { + self.RUNBOOK: "`tx_hash` `consensus.round`", + self.GLOSSARY: "`xrpl.network.id` `ledger_seq`", + self.BUILD_GUIDE: "`service.name` `peer_id`", + self.TESTING: "`rpc_command` `tx.process`", + } + ) + self.assertEqual(tokens, []) + self.assertEqual(report.violations, []) + self.assertTrue(any("E:" in c and "4 file(s)" in c for c in report.checked)) + + def test_skips_when_no_doc_present(self): + tokens, report = _run_rule_e_docs({}) + self.assertEqual(tokens, []) + self.assertEqual(report.checked, []) + self.assertTrue(any("SKIP: E" in s for s in report.skips)) + + def test_exemptions_apply_in_every_discovered_doc(self): + # The L1 resource attrs and perf-iac dotted identities must stay exempt + # in every discovered doc, exactly as in the runbook. + tokens, _ = _run_rule_e_docs( + { + self.GLOSSARY: "`xrpl.network.id` `xrpl.network.type`", + self.BUILD_GUIDE: "`xrpl.work.item` `xrpl.branch` `xrpl.node.role`", + } + ) + self.assertEqual(tokens, []) + + +def _marker(*keys: str) -> str: + """Build an allow-dotted marker naming `keys` (no key = the bare form).""" + body = ": " + ", ".join(keys) if keys else "" + return f"" + + +def _marker_warnings(report): + """The marker-related warnings only. + + A synthetic doc set that does not create the runbook leaves that required + anchor absent, which is warned about in its own right (see + RuleERequiredDocPresence). Filtering those out keeps the marker tests + asserting the marker's own behaviour.""" + return [tuple(w) for w in report.warnings if w[2] != "absent"] + + +class RuleEAllowDottedMarkerParsing(unittest.TestCase): + """`rule_e_allowed_keys` — the marker parser. The key list is what bounds the + exemption, so its parsing (and the marker COUNT, which distinguishes "no + marker" from "a marker that names nothing") is asserted directly.""" + + def test_no_marker_yields_no_keys_and_no_marker_count(self): + self.assertEqual( + chk.rule_e_allowed_keys("plain `xrpl.tx.hash` line"), (set(), 0) + ) + + def test_bare_marker_counts_but_names_no_key(self): + self.assertEqual(chk.rule_e_allowed_keys(_marker()), (set(), 1)) + + def test_single_key(self): + self.assertEqual( + chk.rule_e_allowed_keys(_marker("xrpl.tx.hash")), ({"xrpl.tx.hash"}, 1) + ) + + def test_comma_separated_keys(self): + self.assertEqual( + chk.rule_e_allowed_keys(_marker("xrpl.tx.hash", "xrpl.node.server_state")), + ({"xrpl.tx.hash", "xrpl.node.server_state"}, 1), + ) + + def test_space_separated_and_backticked_keys(self): + self.assertEqual( + chk.rule_e_allowed_keys( + "" + ), + ({"xrpl.tx.hash", "xrpl.peer.id"}, 1), + ) + + def test_no_surrounding_whitespace(self): + self.assertEqual( + chk.rule_e_allowed_keys(""), + ({"xrpl.tx.hash"}, 1), + ) + + def test_two_markers_on_one_line_union(self): + line = _marker("xrpl.tx.hash") + " text " + _marker("xrpl.peer.id") + self.assertEqual( + chk.rule_e_allowed_keys(line), ({"xrpl.tx.hash", "xrpl.peer.id"}, 2) + ) + + def test_prose_around_marker_is_not_absorbed_as_a_key(self): + line = "use `tx_hash`, not `xrpl.tx.hash`. " + _marker("xrpl.tx.hash") + " ok" + self.assertEqual(chk.rule_e_allowed_keys(line), ({"xrpl.tx.hash"}, 1)) + + def test_trailing_comma_does_not_yield_empty_key(self): + self.assertEqual( + chk.rule_e_allowed_keys(""), + ({"xrpl.tx.hash"}, 1), + ) + + +class RuleEAllowDottedMarkerEnforcement(unittest.TestCase): + """The marker exempts ONLY the keys it names, only on its own line. A + blanket line opt-out would let a genuine violation appended to any marked + line ride in silently, which is exactly what these tests forbid.""" + + REFERENCE = GLOSSARY_DOC + + def test_named_key_is_exempt(self): + tokens, report = _run_rule_e_docs( + { + self.REFERENCE: "use `tx_hash`, not `xrpl.tx.hash`. " + + _marker("xrpl.tx.hash") + } + ) + self.assertEqual(tokens, []) + self.assertEqual(report.violations, []) + self.assertEqual(_marker_warnings(report), []) + + def test_unlisted_key_on_marked_line_still_fails(self): + # The hole this closes: one marker must not cover a second dotted key + # someone appends to the line later. + tokens, report = _run_rule_e_docs( + { + self.REFERENCE: "not `xrpl.tx.hash` and also `xrpl.peer.id` " + + _marker("xrpl.tx.hash") + } + ) + self.assertEqual(tokens, ["xrpl.peer.id"]) + self.assertEqual(report.violations[0][0], "E") + self.assertEqual(report.violations[0][1], f"{self.REFERENCE}:1") + self.assertEqual(report.violations[0][3], "underscore, not dotted") + + def test_bare_marker_exempts_nothing_and_warns(self): + tokens, report = _run_rule_e_docs( + {self.REFERENCE: "not `xrpl.tx.hash`. " + _marker()} + ) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + self.assertEqual(report.violations[0][1], f"{self.REFERENCE}:1") + warnings = _marker_warnings(report) + self.assertEqual( + [w[:3] for w in warnings], [("E", f"{self.REFERENCE}:1", "allow-dotted")] + ) + self.assertIn(chk.RULE_E_MARKER_SYNTAX, warnings[0][3]) + + def test_marker_does_not_leak_to_adjacent_lines(self): + tokens, report = _run_rule_e_docs( + { + self.REFERENCE: "| `xrpl.tx.hash` | before |\n" + + "not `xrpl.tx.hash`. " + + _marker("xrpl.tx.hash") + + "\n| `xrpl.tx.hash` | after |" + } + ) + self.assertEqual(tokens, ["xrpl.tx.hash", "xrpl.tx.hash"]) + self.assertEqual( + sorted(v[1] for v in report.violations), + [f"{self.REFERENCE}:1", f"{self.REFERENCE}:3"], + ) + + def test_all_named_keys_exempt_on_a_table_row(self): + # The real-world shape: a "was -> is now" row naming two old + # dotted keys, with both listed in the marker. + tokens, _ = _run_rule_e_docs( + { + self.REFERENCE: "| `xrpl.validation.full`, `xrpl.peer.validation.full`" + " | one bare `full_validation` | " + + _marker("xrpl.validation.full", "xrpl.peer.validation.full") + } + ) + self.assertEqual(tokens, []) + + def test_partially_listed_table_row_fails_on_the_unlisted_key(self): + tokens, report = _run_rule_e_docs( + { + self.REFERENCE: "| `xrpl.validation.full`, `xrpl.peer.validation.full`" + " | one bare `full_validation` | " + _marker("xrpl.validation.full") + } + ) + self.assertEqual(tokens, ["xrpl.peer.validation.full"]) + self.assertEqual(report.violations[0][1], f"{self.REFERENCE}:1") + + def test_key_prefix_does_not_exempt_a_longer_key(self): + # Exemption is an exact token match, not a prefix match. + tokens, _ = _run_rule_e_docs( + {self.REFERENCE: "`xrpl.tx.hash` " + _marker("xrpl.tx")} + ) + self.assertEqual(tokens, ["xrpl.tx.hash"]) + + def test_dotted_prefix_token_can_be_listed_verbatim(self): + # The docs grep for prefixes such as `xrpl.node.` (trailing dot); the + # marker must accept that exact token. + tokens, report = _run_rule_e_docs( + { + self.REFERENCE: "a grep for `xrpl.node.` and `xrpl.peer.` returns " + "nothing " + _marker("xrpl.node.", "xrpl.peer.") + } + ) + self.assertEqual(tokens, []) + self.assertEqual(_marker_warnings(report), []) + + def test_stale_key_warns_but_does_not_fail(self): + tokens, report = _run_rule_e_docs( + {self.REFERENCE: "all clean now. " + _marker("xrpl.tx.hash")} + ) + self.assertEqual(tokens, []) + self.assertEqual(report.violations, []) + self.assertEqual( + _marker_warnings(report), + [ + ( + "E", + f"{self.REFERENCE}:1", + "xrpl.tx.hash", + "allow-dotted key not on this line", + ) + ], + ) + + def test_marker_naming_an_l1_key_is_not_reported_stale(self): + # `xrpl.network.id` is exempt via L1, so a marker naming it is redundant + # but not stale — the key IS on the line. + tokens, report = _run_rule_e_docs( + {self.REFERENCE: "`xrpl.network.id` " + _marker("xrpl.network.id")} + ) + self.assertEqual(tokens, []) + self.assertEqual(_marker_warnings(report), []) + + def test_marker_works_in_every_doc_of_the_set(self): + docs = { + rel: "not `xrpl.tx.hash`. " + _marker("xrpl.tx.hash") for rel in ALL_DOCS + } + tokens, report = _run_rule_e_docs(docs) + self.assertEqual(tokens, []) + self.assertEqual(report.violations, []) + + +class RuleEDocDiscovery(unittest.TestCase): + """`rule_e_docs` — the discovery that replaced a hardcoded doc list. What has + to hold: it finds the real docs in this repo (a rule scanning an empty set + passes without checking anything), it recurses into subdirectories, it never + checks a file twice, and every path it is configured with lies inside a tree + that actually ships.""" + + def test_discovery_finds_more_than_one_doc_in_this_repo(self): + # The vacuous-pass guard, measured against the real tree: one file (or + # none) would mean discovery had collapsed and Rule E was reporting green + # over almost nothing. + found = chk.rule_e_docs(chk.repo_root()) + self.assertGreater(len(found), 1, found) + + def test_discovery_returns_no_duplicates(self): + # Roots may overlap (a root and a subdirectory of it); a file must still + # be checked once, so line numbers are not reported twice. + found = chk.rule_e_docs(chk.repo_root()) + self.assertEqual(len(found), len(set(found))) + + def test_discovery_finds_the_known_telemetry_docs(self): + # Named here as an expectation of DISCOVERY, not as the checker's config: + # if any of these is renamed, discovery still covers it under its new + # name and only this assertion needs updating. + found = set(chk.rule_e_docs(chk.repo_root())) + for rel in (RUNBOOK_DOC, GLOSSARY_DOC, TESTING_DOC): + self.assertIn(Path(rel), found) + + def test_discovery_recurses_into_subdirectories(self): + d = Path(tempfile.mkdtemp()) + try: + nested = d / "docs" / "build" / "deep" + nested.mkdir(parents=True) + (nested / "telemetry.md").write_text("`tx_hash`") + self.assertEqual( + chk.rule_e_docs(d), [Path("docs") / "build" / "deep" / "telemetry.md"] + ) + finally: + shutil.rmtree(d) + + def test_discovery_ignores_non_markdown(self): + d = Path(tempfile.mkdtemp()) + try: + (d / "docs").mkdir() + (d / "docs" / "notes.txt").write_text("`xrpl.tx.hash`") + (d / "docs" / "runbook.md").write_text("`tx_hash`") + self.assertEqual(chk.rule_e_docs(d), [Path("docs") / "runbook.md"]) + finally: + shutil.rmtree(d) + + def test_discovery_tolerates_an_absent_root(self): + # Presence gating applies to the roots too: a tree carrying only one of + # them must not raise, and must still discover that one. + d = Path(tempfile.mkdtemp()) + try: + (d / "docker" / "telemetry").mkdir(parents=True) + (d / "docker" / "telemetry" / "TESTING.md").write_text("`tx_hash`") + self.assertEqual( + chk.rule_e_docs(d), [Path("docker") / "telemetry" / "TESTING.md"] + ) + finally: + shutil.rmtree(d) + + def test_no_configured_path_lies_outside_the_committable_roots(self): + # The check may only be wired to trees that ship. A configured path + # outside them would make Rule E warn about a doc that can never appear + # and leave the matching CI path-trigger dead. + for rel in chk.RULE_E_DOC_ROOTS + chk.RULE_E_REQUIRED_DOCS: + self.assertTrue(_inside_any(rel, COMMITTABLE_ROOTS), rel) + + def test_configured_paths_exist_and_are_version_controlled(self): + # "Committable" is not just a naming claim: git must actually track the + # contents of every configured path in this repo. + root = chk.repo_root() + for rel in chk.RULE_E_DOC_ROOTS: + self.assertTrue((root / rel).is_dir(), rel) + self.assertTrue(_git_tracks(root, rel), rel) + for rel in chk.RULE_E_REQUIRED_DOCS: + self.assertTrue((root / rel).is_file(), rel) + self.assertTrue(_git_tracks(root, rel), rel) + + def test_every_required_doc_is_discoverable(self): + # An anchor outside every root could never be found, so the tripwire + # would fire on every run and stop meaning anything. + for rel in chk.RULE_E_REQUIRED_DOCS: + self.assertTrue(_inside_any(rel, chk.RULE_E_DOC_ROOTS), rel) + self.assertIn(rel, chk.rule_e_docs(chk.repo_root())) + + +class RuleERequiredDocPresence(unittest.TestCase): + """A RULE_E_REQUIRED_DOCS anchor that is not in the tree must be VISIBLE. + Discovery cannot go stale, but it can go quiet — a root emptied of telemetry + docs still yields a clean run — so the anchor's absence is warned about and + counted, while staying non-fatal so a partial branch still passes.""" + + RUNBOOK = RUNBOOK_DOC + + def test_every_required_doc_resolves_in_this_repo(self): + # If the runbook is renamed and RULE_E_REQUIRED_DOCS is not updated, the + # anchor warning would fire on every run; catch it here instead. + root = chk.repo_root() + self.assertEqual( + [ + str(rel) + for rel in chk.RULE_E_REQUIRED_DOCS + if not (root / rel).is_file() + ], + [], + ) + + def test_missing_anchor_is_warned_once(self): + # Discovery finds the glossary, so the rule runs — but the anchor it is + # required to find is gone, which must be said out loud. + tokens, report = _run_rule_e_docs({GLOSSARY_DOC: "`tx_hash`"}) + self.assertEqual(tokens, []) + self.assertEqual( + [tuple(w) for w in report.warnings], + [ + ( + "E", + self.RUNBOOK, + "absent", + "required Rule E doc not in tree (renamed?)", + ) + ], + ) + + def test_ok_line_names_the_absent_anchor_count(self): + tokens, report = _run_rule_e_docs({GLOSSARY_DOC: "`tx_hash`"}) + self.assertEqual(tokens, []) + self.assertTrue( + any( + "1 file(s) checked" in c + and f"1 of {len(chk.RULE_E_REQUIRED_DOCS)} required doc(s) absent" in c + for c in report.checked + ), + report.checked, + ) + + def test_full_doc_set_warns_nothing_and_reports_the_full_count(self): + docs = {rel: "`tx_hash`" for rel in ALL_DOCS} + tokens, report = _run_rule_e_docs(docs) + self.assertEqual(tokens, []) + self.assertEqual(report.warnings, []) + self.assertTrue( + any( + f"{len(ALL_DOCS)} file(s) checked" in c and "absent" not in c + for c in report.checked + ), + report.checked, + ) + + def test_anchor_present_warns_nothing_even_with_other_docs_absent(self): + # Only the ANCHOR is required. A tree carrying just the runbook is a + # legitimate partial branch, not a shrinking doc set. + tokens, report = _run_rule_e_docs({self.RUNBOOK: "`tx_hash`"}) + self.assertEqual(tokens, []) + self.assertEqual(report.warnings, []) + self.assertTrue( + any("1 file(s) checked" in c and "absent" not in c for c in report.checked), + report.checked, + ) + + def test_absent_anchor_is_warned_even_when_l1_is_empty(self): + # The anchor can be verified without an L1 key set, so the visibility + # signal must not depend on the (separately gated) key comparison. + tokens, report = _run_rule_e_docs({GLOSSARY_DOC: "`xrpl.tx.hash`"}, l1_keys=[]) + self.assertEqual(tokens, []) + self.assertEqual(len(report.warnings), len(chk.RULE_E_REQUIRED_DOCS)) + self.assertTrue(any("SKIP: E" in s for s in report.skips)) + + def test_no_anchor_warning_when_discovery_finds_nothing(self): + # Nothing to check at all is already reported by the SKIP line, which + # names the roots it searched, so an anchor warning would be noise. + tokens, report = _run_rule_e_docs({}) + self.assertEqual(tokens, []) + self.assertEqual(report.warnings, []) + roots = ", ".join(str(rel) for rel in chk.RULE_E_DOC_ROOTS) + self.assertEqual( + report.skips, + [f"SKIP: E — no doc-layer file present (no *.md under {roots})"], + ) + + def test_doc_outside_every_root_is_not_discovered(self): + # The counterpart of the roots being committable: markdown parked outside + # them is not part of the L5 layer, so a dotted key there cannot fail the + # build — and equally cannot make the rule look like it checked something. + tokens, report = _run_rule_e_docs({"elsewhere/notes.md": "`xrpl.tx.hash`"}) + self.assertEqual(tokens, []) + self.assertEqual(report.checked, []) + self.assertTrue(any("SKIP: E" in s for s in report.skips)) + + class DslParser(unittest.TestCase): """The makeStr/join/seg:: constexpr DSL resolver — the foundation of the L1 key set. Covers flat, nested, cross-file, alias, and multi-line forms.""" @@ -1068,7 +1661,7 @@ class RuleEReportTuple(unittest.TestCase): (d / "docs").mkdir() (d / "docs" / "telemetry-runbook.md").write_text("`xrpl.tx.hash`") report = chk.Report() - chk.run_rule_e_runbook(d, {"xrpl.network.id"}, report) + chk.run_rule_e_docs(d, {"xrpl.network.id"}, report) self.assertEqual(len(report.violations), 1) rule, _loc, token, expected = report.violations[0] self.assertEqual(rule, "E") @@ -1085,7 +1678,7 @@ class RuleEReportTuple(unittest.TestCase): "`tx_hash` `consensus.round`" ) report = chk.Report() - chk.run_rule_e_runbook(d, {"tx_hash"}, report) + chk.run_rule_e_docs(d, {"tx_hash"}, report) self.assertEqual(report.violations, []) self.assertTrue(any("E:" in c for c in report.checked)) finally: diff --git a/.github/workflows/on-pr.yml b/.github/workflows/on-pr.yml index d48d1fe30b..d119467c0f 100644 --- a/.github/workflows/on-pr.yml +++ b/.github/workflows/on-pr.yml @@ -77,6 +77,16 @@ jobs: .github/workflows/reusable-check-rename.yml .github/workflows/on-pr.yml + # The non-code layers the OTel naming check validates: the docs that + # publish attribute tables (Rule E) and the telemetry stack config — + # collector, Tempo, dashboards (Rules B, C, D). Without these paths a + # docs-only or dashboard-only pull request sets `go=false`, so the + # very layers those rules exist to police would never be checked. + # As with `README.md` below, matching one of these also switches on + # the rest of the workflow; there is a single `go` gate. + docs/** + docker/telemetry/** + # Keep the paths below in sync with those in `on-trigger.yml`. .github/actions/build-deps/** .github/actions/generate-version/** diff --git a/.github/workflows/reusable-check-otel-naming.yml b/.github/workflows/reusable-check-otel-naming.yml index a7af2da8cd..54cab30640 100644 --- a/.github/workflows/reusable-check-otel-naming.yml +++ b/.github/workflows/reusable-check-otel-naming.yml @@ -21,6 +21,13 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + - name: Test the OTel naming checker + # The checker's own unit tests, run before the check itself so a broken + # rule is reported as a broken rule rather than as a naming violation + # (or, worse, as a rule that silently stops flagging anything). + # stdlib `unittest` only: the repo installs no third-party test runner + # for CI, and the checker itself is deliberately dependency-free. + run: python -m unittest discover -s .github/scripts/otel-naming -p 'test_*.py' --verbose - name: Check OTel naming # The script is stdlib-only and reads only files already in the tree; # it enforces each rule only when the layer it needs is present, so it diff --git a/.github/workflows/telemetry-validation.yml b/.github/workflows/telemetry-validation.yml index cff532c3fd..55c81d4465 100644 --- a/.github/workflows/telemetry-validation.yml +++ b/.github/workflows/telemetry-validation.yml @@ -26,25 +26,32 @@ name: Telemetry Validation on: workflow_dispatch: + # NOTE: rpc_rate / rpc_duration / tx_tps / tx_duration have NO effect. + # They are forwarded to run-full-validation.sh, which parses them into + # shell variables and never reads them again — load shape comes entirely + # from --profile and docker/telemetry/workload/workload-profiles.json. + # They are kept (and labelled) rather than removed so existing dispatch + # bookmarks and any saved input sets do not break. To change the load, + # edit or add a profile in workload-profiles.json. inputs: rpc_rate: - description: "RPC load rate (requests per second)" + description: "UNUSED — has no effect. Load shape comes from the workload profile." required: false default: "50" rpc_duration: - description: "RPC load duration (seconds)" + description: "UNUSED — has no effect. Load shape comes from the workload profile." required: false default: "120" tx_tps: - description: "Transaction submit rate (TPS)" + description: "UNUSED — has no effect. Load shape comes from the workload profile." required: false default: "5" tx_duration: - description: "Transaction submit duration (seconds)" + description: "UNUSED — has no effect. Load shape comes from the workload profile." required: false default: "120" run_benchmark: - description: "Run performance benchmarks" + description: "Run performance benchmarks (the only input that changes behaviour)" required: false type: boolean default: false @@ -54,11 +61,19 @@ on: - "pratik/otel-phase*" - "feature/otel-*" - "feature/telemetry-*" + # Keep these globs pointing at paths that actually exist. Two earlier + # entries (include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*) + # matched zero tracked files, so a pure C++ telemetry change never + # triggered this workflow on push — only edits under docker/telemetry/** + # or to this file did. The telemetry sources live in the three telemetry + # module directories below. paths: - ".github/workflows/telemetry-validation.yml" - "docker/telemetry/**" - - "include/xrpl/basics/Telemetry*.h" - - "src/xrpld/app/misc/Telemetry*" + - "include/xrpl/telemetry/**" + - "src/libxrpl/telemetry/**" + - "src/libxrpl/beast/insight/**" + - "src/xrpld/telemetry/**" concurrency: group: telemetry-validation-${{ github.ref }} @@ -141,6 +156,13 @@ jobs: build_type: Release log_verbosity: verbose + # telemetry is passed explicitly even though the CMake option and the + # Conan recipe both default it on. The whole point of this workflow is to + # exercise telemetry, so it should not silently depend on a default it + # does not control: if that default ever flips, every span and metric + # assertion would fail for a reason no log names. Stated here, a build + # without the dependency fails loudly instead, because CMakeLists.txt + # does find_package(opentelemetry-cpp CONFIG REQUIRED) under this option. - name: Configure CMake working-directory: ${{ env.BUILD_DIR }} run: | @@ -148,6 +170,7 @@ jobs: -G Ninja \ -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ + -Dtelemetry=ON \ .. - name: Build xrpld @@ -206,6 +229,11 @@ jobs: TX_DURATION: ${{ github.event.inputs.tx_duration || '120' }} RUN_BENCHMARK: ${{ github.event.inputs.run_benchmark }} run: | + # The four rate/duration flags below are inert (see the + # workflow_dispatch inputs note): run-full-validation.sh parses them + # and never reads them. Load shape comes from the default + # --profile full-validation. They are still passed so the flags stay + # exercised if they are ever wired up. ARGS="--xrpld ${{ env.BUILD_DIR }}/xrpld --skip-loki" ARGS="$ARGS --rpc-rate $RPC_RATE" ARGS="$ARGS --rpc-duration $RPC_DURATION" @@ -234,6 +262,11 @@ jobs: # and `if: failure()` never fires -- which silently skipped these logs on # every failed run, and they are the only record of why a node did not # reach consensus. + # + # stdout.log matters as much as debug.log: a node that dies before its + # log sink opens writes no debug.log at all, so stdout is the only place + # its reason survives. A run that timed out at 4/5 nodes was left + # undiagnosable because that file was not collected. - name: Upload node logs if: always() && steps.validation.outcome != 'success' uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 @@ -241,6 +274,7 @@ jobs: name: xrpld-node-logs path: | /tmp/xrpld-validation/node*/debug.log + /tmp/xrpld-validation/node*/stdout.log /tmp/xrpld-validation/*.log retention-days: 7 if-no-files-found: warn diff --git a/.gitignore b/.gitignore index 5dc3e0ef86..0b4e68af74 100644 --- a/.gitignore +++ b/.gitignore @@ -89,5 +89,6 @@ target/ # clangd cache /.cache -docker/telemetry/workload/__pycache__/ -.claude/ + +# Env. file carrying environmental setup data for local or cloud runs. +.env.* diff --git a/CMakeLists.txt b/CMakeLists.txt index fd3784b66c..86fa97dbde 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -141,10 +141,20 @@ if(rocksdb) endif() # OpenTelemetry distributed tracing (optional). -# When ON, links against opentelemetry-cpp and defines XRPL_ENABLE_TELEMETRY -# so that SpanGuard factory methods produce real OTel spans. -# When OFF (default), all tracing code compiles to no-ops with zero overhead. -# Enable via: conan install -o telemetry=True, or cmake -Dtelemetry=ON. +# When ON, links against opentelemetry-cpp and defines XRPL_ENABLE_TELEMETRY so +# that SpanGuard factory methods produce real OTel spans. +# When OFF, all tracing code compiles to no-ops with zero overhead and +# opentelemetry-cpp is not needed at all. +# +# The value below is temporarily ON so that CI compiles the telemetry code +# paths while this feature is in review. OFF is the intended shipped default; +# flipping it back is tracked as a separate change. Do not rely on the current +# value - select it explicitly with cmake -Dtelemetry=ON|OFF or +# conan install -o telemetry=True|False. +# +# -DXRPL_ENABLE_TELEMETRY=OFF does not turn anything off: that name is only a +# compile definition added below, not a CMake option, so CMake just lists it as +# an unused variable at the end of configuration. option(telemetry "Enable OpenTelemetry tracing" ON) if(telemetry) find_package(opentelemetry-cpp CONFIG REQUIRED) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 81f78e11db..fd00f0e448 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -413,6 +413,29 @@ land in one pull request or several. Run it locally with: python .github/scripts/otel-naming/check_otel_naming.py ``` +### Naming a wrong form in prose (`otel-naming:allow-dotted`) + +The doc rule (E) flags any dotted `` `xrpl..` `` key in the +telemetry docs, because a reader copies those keys straight into a TraceQL or +PromQL query. A doc that _teaches_ the convention, or records a rename, has to be +able to name the wrong form as a counter-example. That mention is opted out with +a marker naming exactly the keys the line is allowed to mention: + +```markdown +Use `tx_hash`, not `xrpl.tx.hash`. + +``` + +- The marker applies to **its own line only**, and exempts **only the keys it + lists** (comma- and/or space-separated, backticks optional). A dotted key on a + marked line that the marker does not name still fails, so an exemption cannot + quietly widen when someone edits the line later. +- A marker with no key list exempts nothing and reports a warning; so does a + marker naming a key the line no longer mentions (a stale exemption). +- Never use it to keep a real attribute table dotted. If the doc publishes a key + an operator is meant to query, fix the key — the marker is for mentions, not + for published attributes. + See [.github/scripts/otel-naming/README.md](.github/scripts/otel-naming/README.md) for the full rule list. diff --git a/OpenTelemetryPlan/02-design-decisions.md b/OpenTelemetryPlan/02-design-decisions.md index ff71e44a80..cbe8151e69 100644 --- a/OpenTelemetryPlan/02-design-decisions.md +++ b/OpenTelemetryPlan/02-design-decisions.md @@ -125,54 +125,79 @@ path in Phase 1b through Phase 5. > **Status column.** This catalog is the design inventory; it is not a > statement of what currently emits. `Live` means the span is present in the -> implemented inventory ([09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-37-spans)), +> implemented inventory ([09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-41-spans)), > which is the authoritative list. `Renamed`/`Split` means the concept shipped > under a different name than planned here. **Not built** means no span is > emitted for it today. > +> **"Not built" is not one thing.** All 14 such entries fall into three cases, and the +> fourth column says which — filing them all as oversights would be wrong: +> +> - **Superseded by metrics or logs (7)** — a deliberate trade-off: the signal is already +> carried by a metric or by a log-derived panel, and a span would add per-event volume +> without adding information. `tx.relay`, `fee.escalate`, `validator.list.fetch`, +> `validator.manifest`, `shamap.sync`, `job.enqueue`, `job.execute`. +> - **Gap (6)** — nothing was decided; they were simply never instrumented. The four +> `peer.*` entries, plus `ledger.replay` and `ledger.delta` — and those last two are the +> sharpest, because they have **no metric substitute at all**. +> - **Deferred (1)** — scheduled work: `amendment.vote` (Phase 11). +> > The four `peer.*` entries are the peer-span coverage gap: only > `peer.proposal.receive` and `peer.validation.receive` exist, so protocol > message send/receive and connection lifecycle are untraced. See > [09 §6.4](./09-data-collection-reference.md#64-peer-span-coverage-gap-not-implemented). +> +> `tx.validate` did ship, but renamed and split three ways: the apply pipeline +> traces `tx.preflight` (stateless checks), `tx.preclaim` (ledger-state checks) +> and `tx.transactor` (application), each stamped with a `stage` attribute. +> Names come from `TxApplySpanNames.h:90,94,99`. The spans are created in two +> different files, not one: `tx.preflight` and `tx.preclaim` come from +> `applySteps.cpp` (`invokePreflight()` at `:211-212`, `invokePreclaim()` at +> `:258-261`, both via the shared `makeStageSpan()` helper at `:89-126`), while +> `tx.transactor` is created in `Transactor::operator()()` +> (`Transactor.cpp:1601-1605`). Query them with +> `name=~"tx\.(preflight|preclaim|transactor)"` — a **single** backslash; RE2 +> reads `\\.` as a literal backslash followed by any character, which matches +> nothing here — never `name="tx.validate"`. -| Span name | Description | Status | -| ------------------------------ | --------------------------------------- | ------------------------------------------------ | -| `tx.receive` | Transaction received from network | Live | -| `tx.validate` | Transaction signature/format validation | **Not built** | -| `tx.process` | Full transaction processing | Live | -| `tx.relay` | Transaction relay to peers | **Not built** | -| `tx.apply` | Apply transaction to ledger | Live | -| `consensus.round` | Complete consensus round | Live | -| `consensus.phase.open` | Open phase - collecting transactions | Live | -| `consensus.phase.establish` | Establish phase - reaching agreement | Renamed `consensus.establish` | -| `consensus.phase.accept` | Accept phase - applying consensus | Renamed `consensus.accept` | -| `consensus.proposal.receive` | Receive peer proposal | Live | -| `consensus.proposal.send` | Send our proposal | Live | -| `consensus.validation.receive` | Receive peer validation | Live | -| `consensus.validation.send` | Send our validation | Live | -| `rpc.request` | HTTP/WebSocket request handling | Split into `rpc.http_request` / `rpc.ws_message` | -| `rpc.command.*` | Specific RPC command (dynamic) | Live | -| `peer.connect` | Peer connection establishment | **Not built** | -| `peer.disconnect` | Peer disconnection | **Not built** | -| `peer.message.send` | Send protocol message | **Not built** | -| `peer.message.receive` | Receive protocol message | **Not built** | -| `ledger.acquire` | Ledger acquisition from network | Live | -| `ledger.build` | Build new ledger | Live | -| `ledger.validate` | Ledger validation | Live | -| `ledger.close` | Close ledger | Renamed `consensus.ledger_close` | -| `ledger.replay` | Ledger replay executed | **Not built** | -| `ledger.delta` | Delta-based ledger acquired | **Not built** | -| `pathfind.request` | Path request initiated | Live | -| `pathfind.compute` | Path computation executed | Live | -| `txq.enqueue` | Transaction queued | Live | -| `txq.apply` | Queued transaction applied | Renamed `txq.apply_direct` / `txq.accept_tx` | -| `fee.escalate` | Fee escalation triggered | **Not built** | -| `validator.list.fetch` | UNL list fetched | **Not built** | -| `validator.manifest` | Manifest update processed | **Not built** | -| `amendment.vote` | Amendment voting executed | **Not built** | -| `shamap.sync` | State tree synchronization | **Not built** | -| `job.enqueue` | Job added to queue | **Not built** | -| `job.execute` | Job execution | **Not built** | +| Span name | Description | Status | Why not built / where the signal lives instead | +| ------------------------------ | --------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `tx.receive` | Transaction received from network | Live | — | +| `tx.validate` | Transaction signature/format validation | Renamed + split → `tx.preflight`, `tx.preclaim`, `tx.transactor` | — | +| `tx.process` | Full transaction processing | Live | — | +| `tx.relay` | Transaction relay to peers | **Not built** | **Superseded by metrics.** Relay volume is carried by the overlay traffic counters (`total_bytes_in/out`, `total_messages_in/out`, per-`TrafficCount` category). Relay is also per-peer fan-out, so one span per relay multiplies by peer count for data the counters already aggregate. | +| `tx.apply` | Apply transaction to ledger | Live | — | +| `consensus.round` | Complete consensus round | Live | — | +| `consensus.phase.open` | Open phase - collecting transactions | Live | — | +| `consensus.phase.establish` | Establish phase - reaching agreement | Renamed `consensus.establish` | — | +| `consensus.phase.accept` | Accept phase - applying consensus | Renamed `consensus.accept` | — | +| `consensus.proposal.receive` | Receive peer proposal | Live | — | +| `consensus.proposal.send` | Send our proposal | Live | — | +| `consensus.validation.receive` | Receive peer validation | Live | — | +| `consensus.validation.send` | Send our validation | Live | — | +| `rpc.request` | HTTP/WebSocket request handling | Split into `rpc.http_request` / `rpc.ws_message` | — | +| `rpc.command.*` | Specific RPC command (dynamic) | Live | — | +| `peer.connect` | Peer connection establishment | **Not built** | **Gap, scoped as its own change** — see [09 §6.4](./09-data-collection-reference.md#64-peer-span-coverage-gap-not-implemented). Adding these changes the 41-family span count and the 40 catalogued in `expected_spans.json`. | +| `peer.disconnect` | Peer disconnection | **Not built** | **Gap.** Partially observable: the aggregate count via the `Overlay.Peer_Disconnects` insight gauge and resource-charge drops via `server_info{metric="peer_disconnects_resources"}`, but not per-reason. Disconnect reasons are only recoverable from `debug.log` (the `log-derived-insights` dashboard). | +| `peer.message.send` | Send protocol message | **Not built** | **Gap.** Of the 13 protocol message families only `mtGET_OBJECTS` has native instrumentation (`getobject_*`); byte/message volume is aggregated by `TrafficCount` category, not traced per message. | +| `peer.message.receive` | Receive protocol message | **Not built** | **Gap.** Same as `peer.message.send`. | +| `ledger.acquire` | Ledger acquisition from network | Live | — | +| `ledger.build` | Build new ledger | Live | — | +| `ledger.validate` | Ledger validation | Live | — | +| `ledger.close` | Close ledger | Renamed `consensus.ledger_close` | — | +| `ledger.replay` | Ledger replay executed | **Not built** | **Gap, no substitute.** `LedgerReplayer.cpp` and `LedgerReplayTask.cpp` contain zero `SpanGuard` uses and no metric covers the replay path. A real hole, not a trade-off. | +| `ledger.delta` | Delta-based ledger acquired | **Not built** | **Gap, no substitute.** `LedgerDeltaAcquire.cpp` contains zero `SpanGuard` uses. The `acquire_*` stats cover whole-ledger acquisition, not the delta path. | +| `pathfind.request` | Path request initiated | Live | — | +| `pathfind.compute` | Path computation executed | Live | — | +| `txq.enqueue` | Transaction queued | Live | — | +| `txq.apply` | Queued transaction applied | Renamed `txq.apply_direct` / `txq.accept_tx` | — | +| `fee.escalate` | Fee escalation triggered | **Not built** | **Superseded by metrics + existing spans.** Escalation state is `txq_metrics{metric=…}` and `load_factor_metrics{metric=…}`; the queueing path that triggers it is already traced by the six `txq.*` spans. An event span would restate a gauge. | +| `validator.list.fetch` | UNL list fetched | **Not built** | **Superseded by metrics.** `validator_health{metric="unl_expiry_days"}`, `{metric="unl_blocked"}` and `{metric="validation_quorum"}` carry the outcome. A fetch span would fire on a slow timer and tell an operator nothing the gauges do not. | +| `validator.manifest` | Manifest update processed | **Not built** | **Superseded by logs.** Per-master-key manifest dispositions are on the `log-derived-insights` dashboard (`ManifestCache` partition, requires `log_level ManifestCache debug`). | +| `amendment.vote` | Amendment voting executed | **Not built** | **Deferred to Phase 11.** `validator_health{metric="amendment_blocked"}` covers the blocked state in the meantime. | +| `shamap.sync` | State tree synchronization | **Not built** | **Superseded by metrics.** Covered by the nine `acquire_*` stats, `nodestore_state{metric=…}` and the five `getobject_*` families. Per-node-fetch spans would be prohibitive volume. | +| `job.enqueue` | Job added to queue | **Not built** | **Superseded by metrics.** `job_queued_total` and `job_queued_us{job_type}` plus the 105 per-job-type `jobq_*` gauges. A span per enqueue is one span per unit of daemon work, for latency the histogram already records exactly. | +| `job.execute` | Job execution | **Not built** | **Superseded by metrics.** `job_started_total`, `job_finished_total`, `job_running_us{job_type}`. Same volume argument as `job.enqueue`. | ### 2.3.3 Attribute Naming Conventions @@ -227,24 +252,36 @@ Resource attributes identify the process and are set once at startup. They use the standard OpenTelemetry semantic conventions plus custom dotted `xrpl.*` keys (the dotted form is reserved for resource scope per §2.3.3). -| Key | Type / value | Description | -| --------------------- | ------------------------------------------------------- | ------------------------------ | -| `service.name` | `"xrpld"` | Standard `SERVICE_NAME` | -| `service.version` | `build_info::getVersionString()` | Standard `SERVICE_VERSION` | -| `service.instance.id` | node public key (base58) | Standard `SERVICE_INSTANCE_ID` | -| `xrpl.network.id` | network id (e.g. 0 for mainnet) | Network identifier | -| `xrpl.network.type` | `"mainnet"` \| `"testnet"` \| `"devnet"` \| `"unknown"` | Network kind | -| `xrpl.node.type` | `"validator"` \| `"stock"` \| `"reporting"` | Node role | -| `xrpl.node.cluster` | cluster name | Cluster name, if clustered | +Five are set, by `Telemetry.cpp:380-387` (tracer resource) and the matching +block in `initMetrics()` (metrics resource); the custom key constants are +`SpanNames.h:117-118`. + +| Key | Type / value | Description | Status | +| --------------------- | -------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `service.name` | `"xrpld"` | Standard `SERVICE_NAME` | Set | +| `service.version` | `build_info::getVersionString()` | Standard `SERVICE_VERSION` | Set | +| `service.instance.id` | node public key (base58), or `[telemetry] service_instance_id` | Standard `SERVICE_INSTANCE_ID` | Set — but the node-key fallback reaches traces only; see [05 §5.1.1](./05-configuration-reference.md) | +| `xrpl.network.id` | network id (e.g. 0 for mainnet) | Network identifier | Set | +| `xrpl.network.type` | `"mainnet"` \| `"testnet"` \| `"devnet"` \| `"unknown"` | Network kind | Set | +| `xrpl.node.type` | `"validator"` \| `"stock"` \| `"reporting"` | Node role | **Not implemented** — no constant, no set-site. Node role is therefore not queryable from a trace. (Dashboards do offer an `$xrpl_node_role` filter, but it matches a Prometheus label stamped by the external perf-iac deployment — `check_otel_naming.py:872` — not by anything in this repo) | +| `xrpl.node.cluster` | cluster name | Cluster name, if clustered | **Not implemented** — no constant, no set-site | + +The collector adds two more resource attributes of its own (`deployment.environment` +and, when the node did not stamp it, `xrpl.network.type`) via the +`resource/tier` processor, and deletes the SDK-injected `telemetry.sdk.*` trio +via `resource/stripsdk`. See [05 §5.5.1](./05-configuration-reference.md). ### 2.4.2 Span Attributes by Category > Span attribute keys use the underscore form from §2.3.3 (shared/qualified > keys are `_`; per-span unique keys are bare). The dotted form > is reserved for the resource attributes in §2.4.1 above. This catalog lists -> the planned attribute set by category; the exact emitted key for each -> implemented span is defined by the `*SpanNames.h` constants, which are the -> single source of truth where the two differ. +> the planned attribute set by category; the exact emitted key **and its type** +> for each implemented span is defined by the `*SpanNames.h` constants and their +> set-sites, which win where the two differ. The types in the tables below are +> the ones originally planned and are **not** all what shipped — `peer_id` is +> the notable case (planned as a base58 string, shipped as an int64). §2.4.3 +> is the implemented view. #### Transaction Attributes @@ -303,15 +340,15 @@ Establish-phase gap fill and cross-node correlation attributes (Phase 4a): #### Peer & Message Attributes -| Key | Type | Description | -| -------------------- | ------- | -------------------------- | -| `peer_id` | string | Peer public key (base58) | -| `peer_address` | string | IP:port | -| `peer_latency_ms` | float64 | Measured latency | -| `peer_cluster` | string | Cluster name if clustered | -| `message_type` | string | Protocol message type name | -| `message_size_bytes` | int64 | Message size | -| `message_compressed` | bool | Whether compressed | +| Key | Type | Description | +| -------------------- | ------- | ------------------------------------------------------------------------- | +| `peer_id` | string | Peer public key (base58) — **planned only; shipped as int64, see §2.4.3** | +| `peer_address` | string | IP:port | +| `peer_latency_ms` | float64 | Measured latency | +| `peer_cluster` | string | Cluster name if clustered | +| `message_type` | string | Protocol message type name | +| `message_size_bytes` | int64 | Message size | +| `message_compressed` | bool | Whether compressed | #### Ledger & Job Attributes @@ -373,22 +410,72 @@ Establish-phase gap fill and cross-node correlation attributes (Phase 4a): ### 2.4.3 Data Collection Summary -The following table summarizes what data is collected by category: +§2.4.2 above is the _planned_ catalogue; this table is the **implemented** one. +Its left column lists the keys of the `attr` namespaces of the `*SpanNames.h` +headers; every key shown has at least one live `attr::` set-site in +non-test code. The right column lists keys this document once claimed were +collected but which have no constant and no set-site at all. -| Category | Attributes Collected | Purpose | -| --------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------- | -| **Transaction** | `tx_hash`, `tx_type`, `tx_result`, `tx_fee`, `ledger_index` | Trace transaction lifecycle | -| **Consensus** | `consensus_round`, `consensus_phase`, `consensus_mode`, `proposers`, `round_time_ms` | Analyze consensus timing | -| **RPC** | `command`, `version`, `rpc_status`, `duration_ms` | Monitor RPC performance | -| **Peer** | `peer_id` (public key), `peer_latency_ms`, `message_type`, `message_size_bytes` | Network topology analysis | -| **Ledger** | `ledger_hash`, `ledger_index`, `close_time`, `ledger_tx_count` | Ledger progression tracking | -| **Job** | `job_type`, `job_queue_ms`, `job_worker` | JobQueue performance | -| **PathFinding** | `pathfind_fast`, `pathfind_search_level`, `pathfind_num_paths`, `pathfind_ledger_index`, `pathfind_num_requests` | Payment path analysis | -| **TxQ** | `txq_queue_depth`, `txq_fee_level`, `txq_eviction_reason` | Queue depth and fee tracking | -| **Fee** | `fee_load_factor`, `fee_escalation_level` | Fee escalation monitoring | -| **Validator** | `validator_list_size`, `validator_list_age_sec` | UNL health monitoring | -| **Amendment** | `amendment_name`, `amendment_status` | Protocol upgrade tracking | -| **SHAMap** | `shamap_type`, `shamap_missing_nodes`, `shamap_duration_ms` | State tree sync performance | +**This table is a category-level roll-up, not the authority.** The +authoritative per-span breakdown — which span carries which attribute — is +[09-data-collection-reference.md §1.2](./09-data-collection-reference.md#12-complete-attribute-inventory-bareunderscore-keys), +and the exact key _spelling_ is owned by the `*SpanNames.h` constants. Where +this table disagrees with either, they win. + +> **Known divergence (documented, not resolved here).** 09 §1.2's Consensus +> subsection lists 47 keys; `include/xrpl/consensus/ConsensusSpanNames.h` +> defines 54 in its `attr` namespace (48 own `makeStr` constants plus 6 +> `using` re-exports of the shared keys in `SpanNames.h`), all 54 with +> set-sites. Five of the difference — `open_duration_ms`, +> `peer_positions_at_close`, `position_hash_prefix`, `prev_ledger_prefix`, +> `disputes_resolved_count` — are emitted but absent from 09 §1.2's consensus +> table; the other two, `proposal_trusted` and `validation_trusted`, are +> documented in 09 §1.2's Peer subsection instead (they are shared keys set on +> both the `peer.*` and the `consensus.*` receive spans — `PeerImp.cpp:1953` +> and `:2027` for the proposal pair, `:2591` and `:2635` for the validation +> pair). Fixing 09 is tracked separately; the Consensus row below lists all 54. + +| Category | Attributes emitted (from `*SpanNames.h`) | Named here but NOT emitted | Purpose | +| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | +| **Transaction** | `tx_hash`, `tx_type`, `ter_result`, `fee`, `sequence`, `current_ledger_seq`, `current_ledger_hash`, `local`, `path`, `suppressed`, `tx_status`, `peer_version`, `peer_id`, `stage`, `applied` | `tx_result` (renamed → `ter_result`), `tx_fee` (→ `fee`), `ledger_index` (→ `current_ledger_seq`), `relay_count`. **`ledger_seq` is not a `tx.*` key**: no `tx.*` span sets it — the receive and apply-stage spans stamp `current_ledger_seq` (`NetworkOPs.cpp:1422`, `PeerImp.cpp:1337`, `Transactor.cpp:1613`, `applySteps.cpp:115`) and, where a view exists, `current_ledger_hash` (`Transactor.cpp:1615`, `applySteps.cpp:121`) | Trace transaction lifecycle | +| **Consensus** | All 54 keys in `ConsensusSpanNames.h`'s `attr` namespace (48 own constants + 6 `using` re-exports), each with a set-site: `consensus_ledger_id`, `consensus_round`, `consensus_round_id`, `consensus_phase`, `consensus_mode`, `consensus_state`, `consensus_result`, `consensus_stalled`, `proposers`, `proposers_finished`, `previous_proposers`, `previous_ledger_seq`, `previous_round_time_ms`, `round_time_ms`, `open_duration_ms`, `quorum`, `proposing`, `is_bow_out`, `trace_strategy`, `converge_percent`, `establish_count`, `tx_count`, `tx_count_open`, `tx_id`, `disputes_count`, `disputes_resolved_count`, `dispute_our_vote`, `dispute_yays`, `dispute_nays`, `agree_count`, `disagree_count`, `threshold_percent`, `avalanche_threshold`, `close_time_threshold`, `have_close_time_consensus`, `close_time_resolution_ms`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `parent_close_time`, `peer_positions_at_close`, `prev_ledger_prefix`, `position_hash_prefix`, `mode_old`, `mode_new`, `validation_sign_time`, `proposal_trusted`, `validation_trusted`; re-exported shared keys `ledger_seq`, `ledger_hash`, `full_validation`, `close_time`, `close_time_correct`, `close_resolution_ms` | — | Analyze consensus timing | +| **RPC** | `command`, `version`, `rpc_role`, `rpc_status`, `request_payload_size`, `is_batch`, `batch_size`, `load_type` | `duration_ms` (span duration is a TraceQL intrinsic — query `duration`), `params` | Monitor RPC performance | +| **Peer** | `peer_id` (**int64**, the process-local `Peer::id_` slot number — not a key of any kind; also set on `tx.receive`), `proposal_trusted`, `validation_trusted`, `ledger_hash`, `full_validation`. (`peer_version` is **not** a peer-span key: the constant lives in `TxSpanNames.h:79` and its only set-site is `PeerImp.cpp:1342` on the `tx.receive` span — see the Transaction row) | `peer_address`, `peer_latency_ms`, `peer_cluster`, `message_type`, `message_size_bytes`, `message_compressed` — the peer-span coverage gap (§2.3.2) | Network topology analysis | +| **Ledger** | `ledger_seq`, `tx_count`, `tx_failed`, `validations`, `acquire_reason`, `timeouts`, `peer_count`, `outcome`, `close_time`, `close_time_correct`, `close_resolution_ms` | `ledger_index` (→ `ledger_seq`), `ledger_tx_count` (→ `tx_count`). `ledger_hash` is a live key, but **no `ledger.*` span sets it** — only `consensus.validation.send` (`RCLConsensus.cpp:977`; that span is the one returned by `createValidationSpan()`, which names `cs::validationSend` at `RCLConsensus.cpp:1365,1373`) and `peer.validation.receive` (`PeerImp.cpp:2573`) do. The `LedgerSpanNames.h:41` `using` alias has zero uses. `consensus.ledger_close` sets **no** hash: its four attributes are `ledger_seq`, `consensus_mode`, `tx_count_open` and `close_time_resolution_ms` (`RCLConsensus.cpp:354-361`) | Ledger progression tracking | +| **gRPC** | `method`, `grpc_role`, `grpc_status` | — | gRPC surface monitoring | +| **Job** | — (no job spans exist) | `job_type`, `job_queue_ms`, `job_worker`. JobQueue is observed via **metrics**, not spans — but by **two disjoint families**, and only one of them has a `job_type` label. See the note below the table | JobQueue performance | +| **PathFinding** | `pathfind_fast`, `pathfind_search_level`, `pathfind_num_paths`, `pathfind_ledger_index`, `pathfind_num_requests`, `pathfind_num_source_assets`, `pathfind_dest_currency`, `pathfind_source_account` (hashed), `pathfind_dest_account` (hashed) | `pathfind_source_currency`, `pathfind_path_count`, `pathfind_cache_hit` | Payment path analysis | +| **TxQ** | `txq_status`, `fee_level_paid`, `required_fee_level`, `queue_size`, `ledger_changed`, `expired_count`, `ter_code`, `retries_remaining`, `num_cleared`, `tx_type`, plus the re-exported shared keys `tx_hash`, `ledger_seq`, `current_ledger_seq`, `current_ledger_hash` | `txq_queue_depth` (→ `queue_size`), `txq_fee_level` (→ `fee_level_paid`), `txq_eviction_reason` | Queue depth and fee tracking | +| **Fee** | — (no `fee.escalate` span, §2.3.2) | `fee_load_factor`, `fee_escalation_level`. Fee escalation is dashboarded from metrics (`fee-market`), not spans | Fee escalation monitoring | +| **Validator** | — (no `validator.*` span, §2.3.2) | `validator_list_size`, `validator_list_age_sec`. UNL health is dashboarded from metrics (`validator-health`) | UNL health monitoring | +| **Amendment** | — (no `amendment.vote` span, §2.3.2) | `amendment_name`, `amendment_status` | Protocol upgrade tracking | +| **SHAMap** | — (no `shamap.sync` span, §2.3.2) | `shamap_type`, `shamap_missing_nodes`, `shamap_duration_ms` | State tree sync performance | + +The right-hand column is the honest gap list: every key in it appears in the +§2.4.2 design catalogue but has **zero set-sites** in the code. Where a rename +happened the live name is given in parentheses; where the concept shipped as a +metric rather than a span that is stated. Do not build a dashboard panel, an +alert rule, or a TraceQL query against anything in that column — the query will +return empty, and (per the PromQL/TraceQL asymmetry) a `=~".*"` matcher on an +absent attribute silently blanks a TraceQL panel while quietly passing in +PromQL. + +> **JobQueue metrics: two families, one label.** The Job row above has no span +> attributes, and the metrics that replace them do **not** all carry a +> `job_type` label. Getting this wrong produces a panel that renders but is +> wrong, so treat the two families as separate query surfaces: +> +> | Family | Where the job type lives | Source | +> | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | +> | Native `XRPL_METRIC_*`: `job_queued_total`, `job_started_total`, `job_finished_total`, `job_queued_us`, `job_running_us` | In a **`job_type` label** | `MetricsRegistry.cpp:360-362` (counters), `:94-95` (histogram names), `:101` (label key) | +> | `beast::insight` `jobq` group: `jobq__waiting` / `_running` / `_deferred` / `_q` | In the **metric name itself** — there is **no** `job_type` label at all | `JobTypeData.h:29-32` (naming contract), `:35-38` (suffixes), `Application.cpp:392` (group) | +> +> **The trap:** `sum by (job_type)(jobq_…)` collapses every job type into a +> single series with an empty `job_type`, because an absent PromQL label is +> equivalent to `""` — the query returns a plausible-looking number rather than +> an error. Aggregate the `jobq_*` family with a name matcher +> (`{__name__=~"jobq_.*_waiting"}`) and reserve `by (job_type)` for the +> `job_*_total` / `job_*_us` family. ### 2.4.4 Privacy & Sensitive Data Policy @@ -400,26 +487,47 @@ OpenTelemetry instrumentation is designed to collect **operational metadata only The following data is explicitly **excluded** from telemetry collection: -| Excluded Data | Reason | -| ----------------------- | ----------------------------------------- | -| **Private Keys** | Never exposed; not relevant to tracing | -| **Account Balances** | Financial data; privacy sensitive | -| **Transaction Amounts** | Financial data; privacy sensitive | -| **Raw TX Payloads** | May contain sensitive memo/data fields | -| **Personal Data** | No PII collected | -| **IP Addresses** | Configurable; excluded by default in prod | +| Excluded Data | Reason | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Private Keys** | Never exposed; not relevant to tracing | +| **Account Balances** | Financial data; privacy sensitive | +| **Transaction Amounts** | Financial data; privacy sensitive | +| **Raw TX Payloads** | May contain sensitive memo/data fields | +| **Personal Data** | No PII collected | +| **IP Addresses** | **Never in spans** — no span sets an address attribute (`peer_address` has zero set-sites); peer spans identify peers by `peer_id`, an int64 process-local slot number. **But the log pipeline is a different story** — see the note below this table | + +> **Peer IPs DO leave the node — via the log pipeline, not via spans.** The +> "IP Addresses" row above is scoped to spans, and only to spans. This same +> document describes a log pipeline (§2.6.5) that carries peer addresses: +> +> 1. `PeerImp`'s constructor logs the peer's `remoteAddress_` — an `IP:port` — +> at `info` severity (`PeerImp.h:837-842`), and other overlay call sites log +> addresses too. These land in the ordinary `debug.log` stream. +> 2. The collector's `filelog` receiver tails exactly that file +> (`otel-collector-config.yaml:38-47`, `include: [/var/log/xrpld/*/debug.log]`) +> and the `logs` pipeline exports it to Loki (`:236-239`). +> +> So a deployment running the shipped stack **does** ship peer IPs off-box, as +> log bodies. There is no attribute to drop and no span-level switch to flip, +> because the IPs are inside free-text log messages rather than in structured +> fields — a `delete` action on an attribute key would not touch them. +> +> **The control points are therefore log-side, not trace-side:** Loki +> retention and access control on the log store; the `filelog` receiver's +> `include` list (dropping it disables log↔trace correlation entirely); or a +> collector-side transform on the log body. Do not describe the telemetry +> pipeline as IP-free without qualifying it to traces. #### Privacy Protection Mechanisms -| Mechanism | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage | -| **Configurable Redaction** | Sensitive fields can be excluded via `[telemetry]` config section | -| **Collector Tail Sampling** | xrpld head sampling is fixed at 1.0 (every span emitted); the collector retains ~10% of non-error traces, reducing stored data exposure | -| **Sampling** | Only 10% of traces recorded by default, reducing data exposure | -| **Local Control** | Node operators have full control over what gets exported | -| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) | -| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector | +| Mechanism | Description | +| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage | +| **Unconditional Redaction** | Account redaction is **not** configurable and cannot be turned off: `redactAccount()` (`Redaction.cpp:14-29`) hashes every **non-empty** address handed to it, with no flag and no bypass (an empty input returns empty — `Redaction.cpp:18-19` — so there is no raw value to leak either way). That is a stronger guarantee than a config switch: there is no insecure-by-default state to misconfigure | +| **Collector Tail Sampling** | **Optional, and OFF in the base stack.** xrpld head sampling is fixed at 1.0 (`Telemetry.h:234` `static constexpr double samplingRatio = 1.0;`), so 100% of traces leave the node. `docker/telemetry/otel-collector-config.yaml` has **no** `tail_sampling` processor either, so the local stack stores 100%. The only shipped policy is in the Grafana Cloud overlay (`otel-collector-config.grafanacloud.yaml:60-67`, wired at `:261`): one `probabilistic` policy at **0.5%**, on the trace-storage branch only so spanmetrics still see every span. Treat sampling as a cost control you opt into — not as a privacy control | +| **Local Control** | Node operators have full control over what gets exported | +| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) | +| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector | #### Account Address Hashing @@ -429,20 +537,40 @@ failure mode. Protection is applied in two independent layers: 1. **SDK-side** (this node): the path-finding RPC handlers call `redactAccount()` (`xrpl::telemetry`, `Redaction.h`) before setting the - `pathfind_source_account` / `pathfind_dest_account` span attributes. The - helper emits the first 16 characters of `sha512Half(address)` as - lowercase hex — deterministic (spans for one account still correlate) - but non-reversible. + `pathfind_source_account` / `pathfind_dest_account` span attributes. For a + non-empty address the helper emits the first 16 characters of + `sha512Half(address)` as lowercase hex — deterministic (spans for one + account still correlate) but non-reversible. An empty address returns empty + rather than the hash of the empty string (`Redaction.cpp:18-19`). 2. **Collector-side** (defense-in-depth): an `attributes/hash` processor in the OpenTelemetry Collector re-hashes those same attributes, so any node that emitted a raw value is still redacted before storage. #### Collector-Level Data Protection -The OpenTelemetry Collector can be configured (via an `attributes` processor) -to hash or redact sensitive attributes before export — for example, hashing -`pathfind_source_account` / `pathfind_dest_account`, deleting `peer_address` -to drop IP addresses, and deleting `params` to redact request parameters. +The shipped base config does exactly one thing here, and it is the +defense-in-depth layer described above: an `attributes/hash` processor +(`otel-collector-config.yaml:105-110`) hashing `pathfind_source_account` and +`pathfind_dest_account`. + +**No `peer_address` or `params` scrubbing rule is needed on the trace pipeline, +and none is shipped.** Earlier drafts prescribed `delete` actions for both. +Neither attribute is ever emitted: `peer_address` has zero set-sites in the code +(peer spans carry `peer_id`, an int64 process-local slot number — not an IP and +not a key), and no span sets a `params` attribute — RPC spans carry `command`, +`version`, `rpc_role`, `rpc_status`, `request_payload_size`, `is_batch`, +`batch_size` and `load_type`, never the request body. Adding delete rules for +absent keys would be harmless but misleading: it would imply the node emits IPs +and request parameters in spans when it does not. + +This says nothing about the **log** pipeline, which is where peer IPs actually +do leave the node (see the note under "Data NOT Collected" above). An +`attributes` processor cannot help there — the addresses are inside free-text +log bodies, not in structured attributes. + +If a future span _does_ introduce an IP-bearing or payload-bearing attribute, +the `attributes` processor is the right place to strip it — and the attribute +should be added to the §2.4 catalogue in the same change. #### Configuration Options for Privacy @@ -555,8 +683,8 @@ flowchart TB proto["message TraceContext {
bytes trace_id = 1; // 16 bytes
bytes span_id = 2; // 8 bytes
uint32 trace_flags = 3;
string trace_state = 4;
}"] end - subgraph jobqueue["JobQueue (Internal Async)"] - job["Context captured at job creation,
restored at execution

class Job {
otel::context::Context
traceContext_;
};"] + subgraph jobqueue["JobQueue / Coroutines (Internal Async)"] + job["CoroAwareContextStorage
(RuntimeContextStorage override)

Per-coroutine context stack,
installed globally at startup.
Job itself carries no context."] end style http fill:#0d47a1,stroke:#082f6a,color:#ffffff @@ -568,7 +696,7 @@ flowchart TB - **HTTP/WebSocket - RPC (blue)**: For client-facing RPC requests, trace context is propagated using the W3C `traceparent` header. This is the standard approach and works with any OTel-compatible client. - **Protocol Buffers - P2P (green)**: For peer-to-peer messages between xrpld nodes, trace context is embedded as a protobuf `TraceContext` message carrying trace_id, span_id, flags, and optional trace_state. -- **JobQueue - Internal Async (red)**: For asynchronous work within a single node, the OTel context is captured when a job is created and restored when the job executes on a worker thread. This bridges the async gap so spans remain linked. +- **JobQueue / Coroutines - Internal Async (red)**: For asynchronous work within a single node, the ambient OTel context follows the coroutine rather than being carried on the work item. `include/xrpl/core/Job.h` has **no** telemetry include and no `traceContext_` member — an earlier draft of this diagram showed one, and that was never built. Instead `xrpl::telemetry::CoroAwareContextStorage` (`include/xrpl/telemetry/CoroAwareContextStorage.h:84`) overrides the SDK's `RuntimeContextStorage` with a per-coroutine context stack, and is installed as the global storage in `Telemetry::start()` (`Telemetry.cpp:416-419`) before the tracer provider and before the first span. That fixes the wrong-thread scope pop across coroutine yield/resume and keeps log↔trace correlation intact. The storage is never reset — tearing it down while spans may still exist is undefined behaviour in the SDK — so it lives for the process lifetime. --- @@ -651,7 +779,10 @@ parent through the active context. ### 2.6.4 Coexistence Strategy -> **Note**: Phase 7 replaces the StatsD bridge with native OTel Metrics SDK export. The diagram below shows the Phase 6 intermediate state. See [Phase7_taskList.md](./Phase7_taskList.md) for the migration design where Beast Insight emits via OTLP instead of StatsD. +> **Note**: Phase 7 **added** a native OTel Metrics export path alongside the +> StatsD bridge; it did not replace it. The diagram below shows the Phase 6 +> state, which is still reachable today via `[insight] server=statsd`. See +> [Phase7_taskList.md](./Phase7_taskList.md) for the design. ```mermaid flowchart TB @@ -681,17 +812,54 @@ flowchart TB - **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/HTTP to a Collector, which then forwards to a trace backend (Tempo). (OTLP/gRPC is future work — §2.2.2.) - **Grafana (red, unified UI)**: All three data streams converge in Grafana, enabling operators to correlate logs, metrics, and traces in a single dashboard. -**Phase 7 target state**: Beast Insight routes to `OTelCollector` (new `Collector` implementation) which exports via OTLP/HTTP to the same collector endpoint as traces. StatsD UDP path becomes a deprecated fallback (`[insight] server=statsd`). See [06-implementation-phases.md §6.8](./06-implementation-phases.md) and [Phase7_taskList.md](./Phase7_taskList.md) for details. +**Phase 7 outcome (as shipped)**: Beast Insight gained an `OTelCollector` +`Collector` implementation that rides the global MeterProvider and exports via +OTLP/HTTP to the same collector as traces. It is selected with +`[insight] server=otel`. -### 2.6.5 Correlation with PerfLog +The three back ends are **co-equal branches of one `if/else` chain** in +`makeCollectorManager()` (`CollectorManager.cpp:37-75`), not a migration path: -Trace IDs can be correlated with existing PerfLog entries for comprehensive -debugging. The design is for `RPCHandler.cpp` to start an `rpc.command.` -span alongside the existing PerfLog `rpcStart`/`rpcFinish`/`rpcError` calls, -extract the span's `trace_id` (when valid), and eventually stamp it onto the -PerfLog entry (a planned `setTraceId` hook) so logs and traces share a key. The -span status is set to OK on success or to error (recording the exception) on -failure. +| `[insight] server=` | Collector | Status | +| ---------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| `otel` | `OTelCollector` | OTLP/HTTP to the OTel Collector — the recommended setting | +| `statsd` | `StatsDCollector` | Unchanged from before Phase 7. **Not deprecated**: no warning is logged, no removal is scheduled, and the code path is not marked legacy | +| absent / anything else | `NullCollector` | **The default.** A node with no `[insight]` section emits no metrics at all | + +Two corrections to earlier drafts, both of which matter operationally: StatsD +is not a "deprecated fallback", and `otel` is not the default — you must set it +explicitly. See [06-implementation-phases.md §6.8](./06-implementation-phases.md), +[Phase7_taskList.md](./Phase7_taskList.md), and +[05 §5.8.6](./05-configuration-reference.md) for which `[insight]` keys are live +under `server=otel` (most are inert). + +### 2.6.5 Correlation with Logs + +**Shipped in Phase 8 — and not the way this section originally planned it.** +The design here was a `setTraceId` hook on PerfLog, fed from the +`rpc.command.` span in `RPCHandler.cpp`. That hook was never built: +`setTraceId` has zero occurrences in **source** — the only hits in the tree are +in these plan documents, describing the design that was dropped — and PerfLog's +JSON output carries no trace ID. + +What shipped instead is broader and needs no per-call-site wiring: the **journal +sink** stamps the IDs onto _every_ log line written while a span is active. +`Logs::format()` (`src/libxrpl/basics/Log.cpp:304-338`, inside +`#ifdef XRPL_ENABLE_TELEMETRY`) reads the thread-local OTel context, and when +the active span context is valid it prefixes the message with +`trace_id=<32 hex> span_id=<16 hex>`. It inspects the context value directly +rather than calling `GetSpan()`, so the common no-span path costs no heap +allocation. + +Because the IDs land in the ordinary `debug.log` stream, correlation is +end-to-end without touching PerfLog: the collector's `filelog` receiver parses +`trace_id`/`span_id` as optional capture groups and ships the lines to Loki, and +Grafana links both directions (Tempo `tracesToLogs` → Loki, Loki derived fields +→ Tempo). Details in [05 §5.8.5](./05-configuration-reference.md). + +RPC spans still exist and still set status (OK on success, error with the +recorded exception on failure) — that part of the original design is intact. +Only the PerfLog-stamping mechanism was replaced. --- diff --git a/OpenTelemetryPlan/03-implementation-strategy.md b/OpenTelemetryPlan/03-implementation-strategy.md index f11af7a929..5eb5750905 100644 --- a/OpenTelemetryPlan/03-implementation-strategy.md +++ b/OpenTelemetryPlan/03-implementation-strategy.md @@ -7,26 +7,54 @@ ## 3.1 Directory Structure -The telemetry implementation follows xrpld's existing code organization pattern: +The telemetry implementation follows xrpld's existing code organization +pattern. The tree below is the current on-disk contents of the three telemetry +directories, and it has three differences from the original design sketch worth +calling out: `TelemetryConfig.h`, `TraceContext.h`, `SpanAttributes.h` and +`TraceContext.cpp` were never created (config structs live inside +`Telemetry.h`, propagation lives in `TraceContextPropagator.h`, and attribute +constants live in the `*SpanNames.h` headers next to their owning class); the +metrics work of Phase 7/9 added a whole second module under +`src/xrpld/telemetry/`, which the sketch predated. ``` -include/xrpl/ -├── telemetry/ -│ ├── Telemetry.h # Main telemetry interface (global singleton) -│ ├── TelemetryConfig.h # Configuration structures -│ ├── TraceContext.h # Context propagation utilities -│ ├── SpanGuard.h # RAII span management with factory methods + discard() -│ ├── DiscardFlag.h # Thread-local discard flag -│ └── SpanAttributes.h # Attribute helper functions +include/xrpl/telemetry/ # libxrpl layer: tracing SDK wrapper +├── Telemetry.h # Interface + Setup config struct + factories +├── SpanGuard.h # RAII span management, factory methods, discard() +├── SpanNames.h # StaticStr/join() + shared span & attr constants +├── DiscardFlag.h # Thread-local discard flag +├── CoroAwareContextStorage.h # RuntimeContextStorage override for coroutines +├── DeterministicIdGenerator.h # trace_id from txHash / prevLedgerHash +├── TraceContextPropagator.h # protobuf TraceContext inject/extract (P2P) +├── TraceContextValidation.h # Validation of peer-supplied trace context +├── Redaction.h # redactAccount() — unconditional address hashing +└── GetObjectMetricNames.h # getobject_* metric name constants -src/libxrpl/ -├── telemetry/ -│ ├── Telemetry.cpp # Implementation + FilteringSpanProcessor -│ ├── TelemetryConfig.cpp # Config parsing -│ ├── TraceContext.cpp # Context serialization -│ └── NullTelemetry.cpp # No-op implementation +src/libxrpl/telemetry/ +├── Telemetry.cpp # TelemetryImpl + FilteringSpanProcessor + initMetrics() +├── TelemetryConfig.cpp # [telemetry] section parsing (makeTelemetrySetup) +├── SpanGuard.cpp # Span/scope guard implementation +├── CoroAwareContextStorage.cpp +├── DeterministicIdGenerator.cpp +├── Redaction.cpp +└── NullTelemetry.cpp # No-op impl — ALWAYS compiled (in-source #ifdef) + +src/xrpld/telemetry/ # xrpld layer: native metrics + tx tracing helpers +├── MetricsRegistry.h / .cpp # Owns the XRPL_METRIC_* instruments + MeterProvider +├── MetricMacros.h # XRPL_METRIC_COUNTER_ADD / _HISTOGRAM_RECORD / ... +├── ValidationTracker.h # Validation-agreement tracking (impl in detail/) +├── detail/ValidationTracker.cpp +├── ConsensusReceiveTracing.h # Peer proposal/validation receive spans +├── PropagationHelpers.h # Context inject/extract call-site helpers +├── TxSpanNames.h # tx.* span + attribute constants +└── TxTracing.h # Transaction span helpers ``` +Per-class span-name headers deliberately live next to their owning class rather +than in `telemetry/` — see `ConsensusSpanNames.h`, `TxApplySpanNames.h`, +`LedgerSpanNames.h`, `RpcSpanNames.h`, `PathFindSpanNames.h`, +`PeerSpanNames.h`, `TxQSpanNames.h`, `GrpcSpanNames.h`. + --- ## 3.2 Implementation Approach @@ -100,13 +128,22 @@ flowchart TB | --------------------- | --------- | ---------------------- | ---------- | | Span creation | 500-1000 | Every traced operation | Low | | Span end | 100-200 | Every traced operation | Low | -| SetAttribute (string) | 80-120 | 3-5 per span | Low | -| SetAttribute (int) | 40-60 | 2-3 per span | Negligible | +| SetAttribute (string) | 80-120 | 3-5 per span (typical) | Low | +| SetAttribute (int) | 40-60 | 2-3 per span (typical) | Negligible | | AddEvent | 100-200 | 0-2 per span | Low | | Context injection | 150-250 | Per outgoing message | Low | | Context extraction | 100-180 | Per incoming message | Low | | GetCurrent context | 10-20 | Thread-local access | Negligible | +> **"3-5 attributes per span" is a typical case, not a bound.** The frequency +> column above describes the median span (`tx.receive`, `rpc.command.*`). A few +> spans are deliberately attribute-rich: `consensus.accept.apply` sets **13** +> attributes (`RCLConsensus.cpp:600-674`), and `consensus.round` / +> `consensus.establish` are of the same order. Use ~15 as the worst case when +> sizing per-span attribute cost and memory; the consensus spans that hit it fire +> once per ~3-second round, so their absolute cost stays in the noise +> (see §3.4.3). + **Source**: Span creation based on OTel C++ SDK `BM_SpanCreation` benchmark (AlwaysOnSampler + SimpleSpanProcessor + InMemoryExporter), median ~1,000 ns on CI hardware. AddEvent includes timestamp read + string copy + vector push + mutex acquisition. Context injection/extraction @@ -120,8 +157,8 @@ confirmed by `BM_SpanCreationWithScope` benchmark delta (~160 ns). %%{init: {'pie': {'textPosition': 0.75}}}%% pie showData "tx.receive (1400ns)" : 1400 - "tx.validate (1200ns)" : 1200 - "tx.relay (1200ns)" : 1200 + "tx.process (1200ns)" : 1200 + "tx.apply (1200ns)" : 1200 "Context inject (200ns)" : 200 ``` @@ -131,9 +168,17 @@ pie showData **Overhead percentage**: 4.0 μs / 200 μs (avg tx processing) = **~2.0%** -> **Breakdown**: Each span (tx.receive, tx.validate, tx.relay) costs ~1,000 ns for creation plus +> **Breakdown**: Each span (tx.receive, tx.process, tx.apply) costs ~1,000 ns for creation plus > ~200-400 ns for 3-5 attribute sets. Context injection is ~200 ns (confirmed by benchmarks). > On production hardware, expect ~2.6 μs total (~1.3% overhead) due to faster span creation (~500-600 ns). +> +> This three-span model predates the apply-pipeline instrumentation. The shipped +> transaction path also emits `tx.preflight`, `tx.preclaim` and `tx.transactor` +> (the spans planned here as `tx.validate`), and never emits `tx.relay`. Scale +> the estimate by span count for a current figure: ~6 spans ≈ 7-8 μs on CI +> hardware, ~4-5 μs on server hardware. The measured end-to-end cost is in +> §3.5.3 (~3-4% throughput at head sampling 1.0), which supersedes this +> bottom-up estimate. ### 3.4.3 Consensus Round Overhead @@ -148,18 +193,26 @@ pie showData > **Why higher**: Each span costs ~1,000 ns creation + ~100-200 ns for 1-2 attributes, totaling ~1,100-1,200 ns. > Context operations remain ~200 ns (confirmed by benchmarks). On production hardware, expect ~24 μs total. +> +> The "1-2 attributes" figure understates the shipped consensus spans, which are +> the attribute-rich ones: `consensus.accept.apply` alone sets 13 +> (`RCLConsensus.cpp:600-674`). Adding ~1 μs per such span still leaves the +> round total under ~40 μs against a ~3 s round, so the conclusion below is +> unaffected. Note also that the `consensus.phase` row covers the shipped names +> `consensus.phase.open`, `consensus.establish` and `consensus.accept` — see +> [02 §2.3.2](./02-design-decisions.md). **Overhead percentage**: 36 μs / 3s (typical round) = **~0.001%** (negligible) ### 3.4.4 RPC Request Overhead -| Operation | Cost (ns) | -| ---------------- | ------------ | -| rpc.request span | ~1200 | -| rpc.command span | ~1100 | -| Context extract | ~250 | -| Context inject | ~200 | -| **TOTAL** | **~2.75 μs** | +| Operation | Cost (ns) | +| ------------------------------------------ | ------------ | +| `rpc.http_request` / `rpc.ws_message` span | ~1200 | +| `rpc.command.*` span | ~1100 | +| Context extract | ~250 | +| Context inject | ~200 | +| **TOTAL** | **~2.75 μs** | > **Why higher**: Each span costs ~1,000 ns creation + ~100-200 ns for attributes (command name, > version, role). Context extract/inject costs are confirmed by OTel C++ benchmarks. @@ -263,12 +316,20 @@ The overhead estimates in Sections 3.3-3.5 are derived from the following source > compression (~60-70% of raw) and batching (amortized headers), ~350 bytes/span is more realistic. > The table uses the conservative estimate for capacity planning. -| Sampling Rate | Spans/sec | Bandwidth | Notes | -| ------------- | --------- | --------- | ---------------- | -| 100% | ~500 | ~250 KB/s | Development only | -| 10% | ~50 | ~25 KB/s | Staging | -| 1% | ~5 | ~2.5 KB/s | Production | -| Error-only | ~1 | ~0.5 KB/s | Minimal overhead | +**Node → collector bandwidth is always the 100% row.** Head sampling is a +`static constexpr` 1.0 (`Telemetry.h:234`) with no config key, so every node +exports every span and the export bandwidth is not tunable from `xrpld.cfg`. + +| Sampling Rate | Spans/sec | Bandwidth | Where it applies | +| --------------------- | --------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| 100% | ~500 | ~250 KB/s | **The only reachable node→collector figure.** Plan capacity against this row | +| 0.5% | ~2.5 | ~1.25 KB/s | Collector→backend only, and only with the Grafana Cloud overlay's `tail_sampling` (`otel-collector-config.grafanacloud.yaml:60-67`) | +| 10% / 1% / error-only | — | — | **Not implemented.** No shipped config produces these ratios; treat them as illustrative of what a tail-sampling policy could do | + +The rows below 100% therefore reduce _storage_ cost at the backend, never the +node's egress. Note also that the shipped 0.5% policy is applied to the +trace-storage branch only, so the spanmetrics-derived RED metrics still see +100% of spans and stay exact. ### 3.6.2 Trace Context Propagation @@ -285,7 +346,26 @@ The overhead estimates in Sections 3.3-3.5 are derived from the following source ### 3.7.1 Sampling Strategies -#### Tail Sampling +#### Head Sampling (node) — fixed, not a decision point + +There is no sampling decision on the node. `samplingRatio` is a +`static constexpr double = 1.0` (`Telemetry.h:234`) and `TelemetryConfig.cpp:139` +records why nothing is parsed: a per-node ratio would let two nodes make +opposite keep/drop decisions for the same distributed trace, yielding partial +traces. The ratio sampler is wrapped in a `ParentBasedSampler` so a span with a +remote parent honours the upstream flag. The only node-local way to drop a span +is the explicit, per-call-site `SpanGuard::discard()`, enforced downstream by +`FilteringSpanProcessor`. + +#### Tail Sampling (collector) — aspirational shape + +The flowchart below is a **design sketch of a multi-policy tail sampler. It is +not what ships.** The base collector config has no `tail_sampling` processor at +all; the Grafana Cloud overlay has exactly one `probabilistic` policy at 0.5% +with no error or latency carve-outs. Read it as a template for a policy you +might write, not as a description of this repo — and note that adding +error/latency policies would need `decision_wait` tuning, since a policy can +only see spans that arrived within that window. ```mermaid flowchart TD @@ -299,13 +379,18 @@ flowchart TD consensus -->|No| slow{"Is Slow?"} slow -->|Yes| sample - slow -->|No| prob{"Random < 10%?"} + slow -->|No| prob{"Probabilistic keep?
(shipped policy: 0.5%)"} prob -->|Yes| sample prob -->|No| drop["DROP"] - style sample fill:#4caf50,stroke:#388e3c,color:#fff - style drop fill:#f44336,stroke:#c62828,color:#fff + style sample fill:#1b5e20,stroke:#0d3d14,color:#fff + style drop fill:#b71c1c,stroke:#7f1d1d,color:#fff + style trace fill:#0d47a1,stroke:#082f6a,color:#fff + style errors fill:#334155,stroke:#1e293b,color:#fff + style consensus fill:#334155,stroke:#1e293b,color:#fff + style slow fill:#334155,stroke:#1e293b,color:#fff + style prob fill:#334155,stroke:#1e293b,color:#fff ``` ### 3.7.2 Batch Tuning Recommendations @@ -318,7 +403,17 @@ flowchart TD ### 3.7.3 Conditional Instrumentation -Instrumentation is gated on two levels. A compile-time feature flag (`XRPL_ENABLE_TELEMETRY`) reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check. +Instrumentation is gated on two levels. A compile-time feature flag reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check. + +> The compile-time gate is the macro `XRPL_ENABLE_TELEMETRY`, but that macro is +> **not** the switch you flip. It is a compile definition added by +> `CMakeLists.txt` (`add_compile_definitions(XRPL_ENABLE_TELEMETRY)`) when the CMake option `telemetry` is ON. +> That option is declared ON today (`option(telemetry "Enable OpenTelemetry tracing" ON)`) +> only so that CI compiles the instrumented build while the telemetry branches are +> in review; **OFF is the intended default once merged**, flipped in a separate +> change. Select the value explicitly instead of relying on the default: +> `-Dtelemetry=ON|OFF` (CMake) or `-o telemetry=True|False` (Conan). See +> [05 §5.4.2](./05-configuration-reference.md). --- @@ -372,39 +467,78 @@ quadrantChart ### 3.9.4 Architectural Impact Assessment -| Aspect | Impact | Justification | -| -------------------- | ------- | -------------------------------------------------------------------------------- | -| **Data Flow** | Minimal | Read-only instrumentation; no modification to consensus or transaction data flow | -| **Threading Model** | Minimal | Context propagation uses thread-local storage (standard OTel pattern) | -| **Memory Model** | Low | Bounded queues prevent unbounded growth; RAII ensures cleanup | -| **Network Protocol** | Low | Optional fields in protobuf (high field numbers); backward compatible | -| **Configuration** | None | New config section; existing configs unaffected | -| **Build System** | Low | Optional CMake flag; builds work without OpenTelemetry | -| **Dependencies** | Low | OpenTelemetry SDK is optional; null implementation when disabled | +| Aspect | Impact | Justification | +| -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Data Flow** | Minimal | Read-only instrumentation; no modification to consensus or transaction data flow | +| **Threading Model** | Minimal | Context propagation uses thread-local storage (standard OTel pattern) | +| **Memory Model** | Low | Bounded queues prevent unbounded growth; RAII ensures cleanup | +| **Network Protocol** | Low | Optional fields in protobuf (high field numbers); backward compatible | +| **Configuration** | None | New config section; existing configs unaffected | +| **Build System** | Low | A single CMake option (`telemetry`) selects the whole feature in or out, and builds work either way (`-Dtelemetry=ON` / `-Dtelemetry=OFF`). It is declared ON today only so CI compiles the instrumented paths; **OFF is the intended default once merged**, so the shipped build is opt-in | +| **Dependencies** | Medium | `opentelemetry-cpp/1.28.0` is a **conditional** requirement, never a hard one: `conanfile.py:152-153` adds it only `if self.options.telemetry`, and `:238-239` adds the matching `libxrpl` component requirement the same way. The option's declared default is `True` today (`conanfile.py:59`), so a default `conan install` does resolve it; with `-o telemetry=False` it never enters the graph and the null implementation supplies the factory | ### 3.9.5 Backward Compatibility -| Compatibility | Status | Notes | -| --------------- | ------- | ----------------------------------------------------- | -| **Config File** | ✅ Full | New `[telemetry]` section is optional | -| **Protocol** | ✅ Full | Optional protobuf fields with high field numbers | -| **Build** | ✅ Full | `XRPL_ENABLE_TELEMETRY=OFF` produces identical binary | -| **Runtime** | ✅ Full | `enabled=0` produces zero overhead | -| **API** | ✅ Full | No changes to public RPC or P2P APIs | +| Compatibility | Status | Notes | +| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Config File** | ✅ Full | New `[telemetry]` section is optional | +| **Protocol** | ✅ Full | Optional protobuf fields with high field numbers | +| **Build** | ✅ Full | `-Dtelemetry=OFF` (or `-o telemetry=False`) produces a binary with all tracing compiled out, whatever the option's declared default happens to be. **Not** `-DXRPL_ENABLE_TELEMETRY=OFF`, which does not disable anything — it is not a CMake option, only a compile definition that `CMakeLists.txt:152` adds inside the `if(telemetry)` block. CMake does flag it (`Manually-specified variables were not used by the project`) at the end of configuration, so it is not literally silent — but the warning is easy to scroll past and the resulting binary still has telemetry compiled in. See [05 §5.4.2](./05-configuration-reference.md) | +| **Runtime** | ✅ Full | `enabled=0` produces zero overhead | +| **API** | ✅ Full | No changes to public RPC or P2P APIs | ### 3.9.6 Rollback Strategy If issues are discovered after deployment: -1. **Immediate**: Set `enabled=0` in config and restart (zero code change) -2. **Quick**: Rebuild with `XRPL_ENABLE_TELEMETRY=OFF` +1. **Immediate**: Set `enabled=0` in `[telemetry]` and restart (zero code change). + Also set `[insight] server=` to something other than `otel` if metrics must + stop too — `enabled=0` governs tracing, and the metrics pipeline is selected + separately ([02 §2.6.4](./02-design-decisions.md)). +2. **Quick**: Rebuild with `-Dtelemetry=OFF` (CMake) or `-o telemetry=False` + (Conan). Pass the flag explicitly — an omitted flag resolves to the option's + declared default, which is ON today and OFF once the feature is merged; a + build that already has telemetry off needs no rebuild at all. + **Do not use `-DXRPL_ENABLE_TELEMETRY=OFF`** — it is not a CMake option, so + it is ignored (CMake reports it under `Manually-specified variables were not +used by the project`) and the rebuilt binary still has telemetry compiled in. + This step also drops the `opentelemetry-cpp` dependency, so expect a full + rebuild rather than an incremental one. 3. **Complete**: Revert telemetry commits (clean separation makes this easy) ### 3.9.7 Code Change Examples **Minimal RPC Instrumentation (Low Intrusiveness):** Instrumenting an RPC handler adds roughly 3-4 lines: one macro to start the span and one or two `setAttribute` calls (command name, status). The span ends automatically via RAII, so the existing control flow — process the request, send the result — is untouched. -**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires storing the active context in a new member variable (`currentRoundContext_`) at round start. The existing round logic itself remains unchanged. +**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires **four** new member variables on the adaptor rather than the single `currentRoundContext_` this section originally sketched (`RCLConsensus.h:103,113,123,143`): + +- `std::optional roundSpan_` (`:103`) — the round span + itself. It is **created and ended in one place**, `startRoundTracing()`: the + previous round's guard is released at `RCLConsensus.cpp:1288-1289` + (`if (roundSpan_) roundSpan_.reset();`) and the new one is emplaced a few + lines later — at `:1306` or `:1310` on the `"attribute"` strategy, at `:1319` + on the default `"deterministic"` one. `preStartRound()` does not create it; it + calls `startRoundTracing()` at `:1229`. There is no `reset()` method — the + span simply lives until the next round begins. A `SpanGuard` owns no + thread-local scope, so emplacing and resetting on different job workers is + safe. +- `telemetry::SpanContext roundSpanContext_` (`:113`) — a lightweight value-type + snapshot, captured at the end of `startRoundTracing()` (`:1350`). Child spans + link through this, not through an ambient parent, so code running on another + worker (e.g. `createValidationSpan()` on `jtACCEPT`) never touches + `roundSpan_` cross-thread. +- `telemetry::SpanContext prevRoundSpanContext_` (`:123`) — the prior round's + context, saved at `:1282` **before** the new span overwrites + `roundSpanContext_`, so the new round span can carry a follows-from link and + consecutive rounds stay navigable. +- `telemetry::SpanContext acceptSpanContext_` (`:143`) — the current round's + accept-span context, set at `:544` and cleared at `:1286` on each new round. + `createValidationSpan()` prefers it as the parent and falls back to + `roundSpanContext_` (`:1363-1373`), so a stale value must not survive into the + next round. + +The split is the point: the guard is owned by one thread, the contexts are +copied freely. The existing round logic itself remains unchanged. --- diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index cc3a83b347..e3b8ae5fe6 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -11,43 +11,115 @@ ### 5.1.1 Configuration File Section -The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows. +The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). Transaction trace IDs are always deterministic (`trace_id = txHash[0:16]`); there is no strategy switch for the transaction path. The full option reference follows. + +> **`service_instance_id` is effectively required for `beast::insight` +> metrics — and only for those.** Three producers resolve the instance id +> independently, and exactly one of them lacks a node-key fallback: +> +> | Producer | Resource built by | Unset `service_instance_id` yields | +> | ------------------------------------------- | -------------------------------------------- | ---------------------------------------------- | +> | Traces (and therefore all `span_*` metrics) | `Telemetry::start()` | Base58 node public key | +> | Native `XRPL_METRIC_*` (`MetricsRegistry`) | `MetricsRegistry::initExporterAndProvider()` | Base58 node public key | +> | `beast::insight` (`[insight] server=otel`) | `TelemetryImpl` **constructor** | **`service.instance.id` absent** — no fallback | +> +> - **Traces**: the tracer resource is built in `Telemetry::start()` +> (`Telemetry.cpp:380-387`), which runs after `ApplicationImp::setup()` has +> called `setServiceInstanceId()` (`Application.cpp:1323`) with the Base58 +> node public key. An unset key therefore still yields the node key. The +> `spanmetrics` connector derives `span_calls_total` / +> `span_duration_milliseconds_*` from those spans, so span metrics inherit +> the correct id too. +> - **Native `XRPL_METRIC_*` metrics** build their **own** MeterProvider +> resource in `MetricsRegistry::initExporterAndProvider()` +> (`MetricsRegistry.cpp:280`, `:296-304`, provider created at `:339`), and +> `ApplicationImp::startTelemetry()` supplies the id with an explicit node-key +> fallback (`Application.cpp:1674-1679`: read the config key, and +> `if (instanceId.empty() && nodeIdentity_)` substitute +> `toBase58(TokenType::NodePublic, …)`). By then `setup()` has resolved +> `nodeIdentity_` (`Application.cpp:1315`), so these metrics carry the node +> key even with the config key unset. +> - **`beast::insight` metrics** are the exception. They use the **global** +> MeterProvider, whose resource is built in the `TelemetryImpl` +> **constructor** (`Telemetry.cpp:321-338`, `initMetrics()` at `:447`), +> because insight instruments are created eagerly in subsystem constructors +> and would otherwise bind to the noop provider forever. At that point +> `serviceInstanceId` is still `""` (`Application.cpp:348` passes an empty +> node key), and the code comment at `Telemetry.cpp:333-336` states plainly +> that the later setter "cannot change this immutable resource". Worse, +> `initMetrics()` sets the attribute **unconditionally** +> (`Telemetry.cpp:488`), so the resource carries `service.instance.id=""` +> rather than omitting it — whereas `MetricsRegistry` guards the same write +> with `if (!instanceId.empty())` (`MetricsRegistry.cpp:302-303`). +> +> Result: with `service_instance_id` unset, `beast::insight` metrics — and only +> those — export with an empty `service.instance.id`. Every shipped Grafana +> dashboard filters on `service_instance_id=~"$node"`, so **insight-backed +> panels** lose their per-node dimension; span-metric and `XRPL_METRIC_*` +> panels are unaffected. Set the key explicitly on any node whose insight +> metrics are dashboarded. +> +> **Known issue.** The asymmetry is a defect, not a design: `MetricsRegistry` +> already demonstrates the node-key fallback that the global provider needs. +> A fix would have to resolve the node identity before `TelemetryImpl` is +> constructed, or make the insight metrics use a late-built provider. ### 5.1.2 Configuration Options Summary -| Option | Type | Default | Description | -| -------------------------- | ------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------- | -| `enabled` | bool | `false` | Enable/disable telemetry | -| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint | -| `use_tls` | bool | `false` | Enable TLS for exporter connection | -| `tls_ca_cert` | string | `""` | Path to CA certificate file | -| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; requires `use_tls=1`; empty = one-way TLS | -| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; requires `use_tls=1`; required when the cert is set | -| `batch_size` | uint | `512` | Spans per export batch | -| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) | -| `max_queue_size` | uint | `2048` | Maximum queued spans | -| `trace_transactions` | bool | `true` | Enable transaction tracing | -| `trace_consensus` | bool | `true` | Enable consensus tracing | -| `trace_rpc` | bool | `true` | Enable RPC tracing | -| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) | -| `trace_ledger` | bool | `true` | Enable ledger tracing | -| `tx_trace_strategy` | string | `"deterministic"` | TX trace ID strategy: `"deterministic"` (trace_id = txHash[0:16]) or `"attribute"` (random) | -| `consensus_trace_strategy` | string | `"deterministic"` | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random) | -| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics | -| `service_instance_id` | string | `` | Instance identifier | +| Option | Type | Default | Description | +| -------------------------- | ------ | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | bool | `false` | Enable/disable telemetry | +| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint for **traces** | +| `metrics_endpoint` | string | `http://localhost:4318/v1/metrics` | OTLP/HTTP collector endpoint for the native metrics pipeline (`MetricsRegistry`). Read in `Application.cpp:1670` | +| `use_tls` | bool | `false` | Enable TLS for exporter connection | +| `tls_ca_cert` | string | `""` | Path to CA certificate file | +| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; requires `use_tls=1`; empty = one-way TLS | +| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; requires `use_tls=1`; required when the cert is set | +| `batch_size` | uint | `512` | Spans per export batch | +| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) | +| `max_queue_size` | uint | `2048` | Maximum queued spans | +| `trace_transactions` | bool | `true` | Enable transaction tracing | +| `trace_consensus` | bool | `true` | Enable consensus tracing | +| `trace_rpc` | bool | `true` | Enable RPC tracing | +| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) | +| `trace_ledger` | bool | `true` | Enable ledger tracing | +| `consensus_trace_strategy` | string | `"deterministic"` | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random). Parsed at `TelemetryConfig.cpp:155-156`, consumed at `RCLConsensus.cpp:1291,1296`. **Not validated** — see the note below | +| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics | +| `service_instance_id` | string | node public key (base58) | Instance identifier (`service.instance.id`). Traces, span metrics and native `XRPL_METRIC_*` metrics all fall back to the node key; **`beast::insight` metrics do not** — see the note in §5.1.1 | + +**`consensus_trace_strategy` is not validated.** `TelemetryConfig.cpp:155-156` +copies the raw string into `Setup::consensusTraceStrategy` without checking it +against an allowed set, and the only comparison in the code is +`strategy == "attribute"` (`RCLConsensus.cpp:1296`). Any unrecognised value — +including a typo — silently takes the deterministic branch with no log warning. +The two accepted values are documented at `include/xrpl/telemetry/Telemetry.h:287-292`. + +**Not a config key — deterministic transaction trace IDs are unconditional.** +Earlier drafts of this document listed a `tx_trace_strategy` option +(`"deterministic"` \| `"attribute"`). No such key exists: `TelemetryConfig.cpp` +parses no transaction-strategy key, and the transaction trace ID is always +derived from the transaction hash. Only the **consensus** path has a +switchable strategy. **Planned (not yet implemented)**: the following options appear in the design -documents but are not parsed by `TelemetryConfig.cpp` in Phase 1b and later -phases. They will be added as the corresponding subsystems are instrumented: +documents but are not parsed by `TelemetryConfig.cpp`. They will be added as +the corresponding subsystems are instrumented: -| Option | Planned Phase | Purpose | -| -------------------------- | ------------- | ----------------------------------------------------------------------- | -| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC | -| `trace_pathfind` | Phase 2 | Path computation tracing toggle | -| `trace_txq` | Phase 3 | Transaction queue tracing toggle | -| `trace_validator` | Future | Validator list / manifest update tracing | -| `trace_amendment` | Future | Amendment voting tracing | -| `consensus_trace_strategy` | Phase 4 | Trace ID strategy for consensus rounds (`deterministic` \| `attribute`) | +| Option | Planned Phase | Purpose | +| ----------------- | ------------- | ---------------------------------------- | +| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC | +| `trace_pathfind` | Phase 2 | Path computation tracing toggle | +| `trace_txq` | Phase 3 | Transaction queue tracing toggle | +| `trace_validator` | Future | Validator list / manifest update tracing | +| `trace_amendment` | Future | Amendment voting tracing | + +> **`exporter` is not read, so do not set it.** Both shipped sample configs +> (`docker/telemetry/xrpld-telemetry.cfg`, +> `docker/telemetry/xrpld-telemetry-mainnet.cfg`) used to carry +> `exporter=otlp_http`; the line had no effect and has since been replaced with +> a comment saying so. OTLP/HTTP is the only transport that exists (§2.2.1), and +> `endpoint` / `metrics_endpoint` are the only transport knobs, until the §2.2.2 +> gRPC work lands. --- @@ -55,7 +127,16 @@ phases. They will be added as the corresponding subsystems are instrumented: > **TxQ** = Transaction Queue -The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2). +The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.valueOr(...)`. It takes `serviceInstanceId` from the `nodePublicKey` argument when the key is absent, applies one unconditional `endpoint` default (`dflt::endpoint`, `TelemetryConfig.cpp:61`, used at `:108`) — the parser has no notion of exporter type — and leaves the sampling ratio at its fixed 1.0 default (a `static constexpr` member, so there is nothing to parse; `TelemetryConfig.cpp:139`, `Telemetry.h:234`). It also rejects two contradictory mTLS configurations outright (`tls_client_cert` without `tls_client_key`, and either without `use_tls=1`) rather than failing open at handshake time. + +`metrics_endpoint` is deliberately **not** handled here: it is read separately in `ApplicationImp::startTelemetry()` (`Application.cpp:1670`) and passed to `MetricsRegistry::start()`. Note the consequence — the two metric exporters resolve their URL differently: + +| Metric source | Exporter built by | URL comes from | +| ------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------- | +| `beast::insight` (`[insight] server=otel`) | `Telemetry::initMetrics()` (global provider) | `endpoint` with a trailing `/v1/traces` rewritten to `/v1/metrics` | +| Native `XRPL_METRIC_*` (`MetricsRegistry`) | `MetricsRegistry::initExporterAndProvider()` | `metrics_endpoint`, defaulting to `http://localhost:4318/v1/metrics` | + +Setting a non-default `endpoint` therefore moves the insight metrics with it, but leaves the native metrics on localhost unless `metrics_endpoint` is set too. --- @@ -68,6 +149,13 @@ The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` > resolved later in `setup()`. The `Telemetry` object is therefore > constructed with an empty `serviceInstanceId` and patched via > `setServiceInstanceId()` once `setup()` has called `getNodeIdentity()`. +> **This patch reaches traces only.** The **global** MeterProvider resource — +> the one `beast::insight` metrics use — is already frozen by then (§5.1.1), so +> those metrics keep whatever `service_instance_id` the config supplied (`""` +> if it supplied none). Native `XRPL_METRIC_*` metrics do not go through this +> patch at all: `startTelemetry()` re-reads the config key and applies its own +> node-key fallback when building `MetricsRegistry`'s separate resource +> (`Application.cpp:1674-1679`). `ApplicationImp` (in `src/xrpld/app/main/Application.cpp`) owns a `std::unique_ptr telemetry_`. It is built in the member initializer list via `makeTelemetry(makeTelemetrySetup(...))` with an empty `serviceInstanceId`, then patched in `setup()` by calling `setServiceInstanceId()` with the Base58 node public key (unless the user supplied a custom `service_instance_id`). `start()` and `run()` forward to `telemetry_->start()` / `telemetry_->stop()`, and `getTelemetry()` returns the owned instance. @@ -87,13 +175,72 @@ The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` > **OTLP** = OpenTelemetry Protocol -### 5.4.1 Find OpenTelemetry Module +### 5.4.1 Locating the OpenTelemetry SDK -A `cmake/FindOpenTelemetry.cmake` module locates the OpenTelemetry C++ SDK. It first tries `find_package(opentelemetry-cpp CONFIG)`, aliasing the imported targets `OpenTelemetry::api`, `OpenTelemetry::sdk`, and `OpenTelemetry::otlp_grpc_exporter`, and falls back to `pkg-config` when no CMake config package is present. +> **Superseded design.** Earlier drafts described a hand-written +> `cmake/FindOpenTelemetry.cmake` module that aliased `OpenTelemetry::api`, +> `OpenTelemetry::sdk` and `OpenTelemetry::otlp_grpc_exporter` with a +> `pkg-config` fallback. That module was never written — it exists in no +> commit — and the aliasing approach it described does not work with the +> package the build actually consumes. + +The SDK is located by the Conan-generated CMake config package, nothing else: + +- `CMakeLists.txt` — `find_package(opentelemetry-cpp CONFIG REQUIRED)`, + guarded by the `telemetry` option (§5.4.2). The dependency itself is + declared in `conanfile.py:153` (`opentelemetry-cpp/1.28.0`), also guarded — + `requirements()` adds it only `if self.options.telemetry` (`:152`), so with + the option off the package never enters the dependency graph. +- Linking goes through the **umbrella** target + `opentelemetry-cpp::opentelemetry-cpp`, never the per-component targets. + `cmake/XrplCore.cmake:221-225` and `:83-91` record why: the Conan package + under-declares its inter-component dependencies, so naming `::api` / `::sdk` + individually produces the wrong static-link order and fails at executable + link time. The umbrella target supplies both the trace and metrics + components with the correct ordering. ### 5.4.2 CMakeLists.txt Changes -The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `OFF`). When enabled, it runs `find_package(OpenTelemetry REQUIRED)`, defines the `XRPL_ENABLE_TELEMETRY` compile flag, and builds the `xrpl_telemetry` library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op `NullTelemetry.cpp` so call sites compile unchanged. +The build flag is `telemetry`: + +``` +option(telemetry "Enable OpenTelemetry tracing" ON) # top-level CMakeLists.txt +``` + +The declared value is ON **temporarily**, so that CI compiles the telemetry code +paths while the feature branches are in review. **OFF is the intended default +once merged**, and the flip is a separate change. Set the value explicitly +rather than relying on the default: + +| To … | Use (CMake) | Use (Conan) | +| ------------------------- | ----------------- | -------------------- | +| Build telemetry in | `-Dtelemetry=ON` | `-o telemetry=True` | +| Build it out (all no-ops) | `-Dtelemetry=OFF` | `-o telemetry=False` | + +When the option is ON, the guarded block below it runs +`find_package(opentelemetry-cpp CONFIG REQUIRED)` and adds the +**compile definition** `XRPL_ENABLE_TELEMETRY`. + +> **`XRPL_ENABLE_TELEMETRY` is not a CMake option.** It is only ever _added_ +> as a compile definition by `add_compile_definitions(XRPL_ENABLE_TELEMETRY)` in that same block. Passing +> `-DXRPL_ENABLE_TELEMETRY=OFF` on the CMake command line disables **nothing** — +> it defines an unused cache variable and telemetry stays compiled in. CMake does +> report it, at the end of configuration under `Manually-specified variables were +not used by the project`, so it is not literally silent — but that line is easy +> to scroll past. Any procedure that relies on it (including the rollback path in +> [§3.9.6](./03-implementation-strategy.md)) must use `-Dtelemetry=OFF`. + +The target is `xrpl.libxrpl.telemetry`, created by `add_module(xrpl telemetry)` +at `cmake/XrplCore.cmake:231` from `include/xrpl/telemetry/` + +`src/libxrpl/telemetry/`. There is no `xrpl_telemetry` target. + +Selection between the real and the no-op implementation is an **in-source +`#ifdef`, not a source swap**: `NullTelemetry.cpp` is compiled into the target +unconditionally (see its header comment, `NullTelemetry.cpp:1-12`). It provides +the `makeTelemetry()` factory when `XRPL_ENABLE_TELEMETRY` is undefined; when +the macro is defined, `Telemetry.cpp` provides the factory instead and +`NullTelemetry`'s virtuals only serve as noop tracer/span fallbacks. Call sites +compile unchanged either way. --- @@ -105,13 +252,101 @@ The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default ` The authoritative collector config lives in the repo at `docker/telemetry/otel-collector-config.yaml` (with Tempo backend config in `docker/telemetry/tempo.yaml`). The sections below summarize the development and production shapes of that pipeline. -### 5.5.1 Development Configuration +### 5.5.1 Development / Base Configuration -The development collector enables an OTLP receiver on both gRPC (`0.0.0.0:4317`) and HTTP (`0.0.0.0:4318`), a single `batch` processor (1s timeout, batch size 100), and two exporters: a `logging` exporter for console debugging and `otlp/tempo` (insecure) for trace visualization. The single `traces` pipeline wires receiver → batch → both exporters. +`docker/telemetry/otel-collector-config.yaml` is the base config used by the +local stack and by CI. It carries **three** pipelines, not one: + +| Pipeline | Receivers | Processors | Exporters | +| --------- | --------------------- | ---------------------------------------------------------------- | ------------------------------------ | +| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp/tempo`, `spanmetrics` | +| `metrics` | `otlp`, `spanmetrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` | +| `logs` | `filelog` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlphttp/loki` | + +Component detail: + +- **Receivers.** `otlp` on gRPC `0.0.0.0:4317` and HTTP `0.0.0.0:4318` (both + traces and native metrics arrive on 4318). `filelog` tails + `/var/log/xrpld/*/debug.log` and runs a `regex_parser` that lifts + `timestamp`, `partition`, `severity` and the optional `trace_id`/`span_id` + emitted by the journal sink (§5.8.5). +- **Processors.** `batch` (1s timeout, `send_batch_size: 100`); + `resource/tier` (`action: upsert` on `deployment.environment`, `action: insert` on + `xrpl.network.type` only when absent); `resource/stripsdk` (drops the + `telemetry.sdk.*` attributes); `resource/logs` (`action: upsert` on + `service.name` and `job` — only the former becomes a Loki stream label, see + the known issue in §5.8.5); `attributes/hash` (hashes + `pathfind_source_account` and `pathfind_dest_account`). +- **Connector.** `spanmetrics` with `namespace: "span"` + (`otel-collector-config.yaml:114`) — this is why the derived RED metrics are + `span_calls_total` / `span_duration_milliseconds_*`. The connector's own + default namespace is **empty**, so without this setting the names would be + the bare `calls_total` / `duration_milliseconds_*`. The + `traces_spanmetrics_*` family is **not** the connector's default and is not + produced here at all — it comes from a different producer, Tempo's + `metrics_generator` `span-metrics` processor (`tempo.yaml:75`), whose + `remote_write` is commented out in this repo (see §5.8.6). Histogram + `unit: ms` + with sub-millisecond buckets from `0.01ms`, plus explicit `2s`–`30s` + boundaries for consensus and `ledger.acquire`. ~25 low-cardinality + dimensions are promoted to labels (`command`, `rpc_status`, `tx_type`, + `ter_result`, `stage`, `consensus_mode`, `outcome`, …). +- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp/tempo` + (`tempo:4317`, `tls.insecure: true`), `otlphttp/loki` + (`http://loki:3100/otlp` — Loki 3.x native OTLP; the old `loki` exporter was + removed in collector-contrib v0.147.0), and `prometheus` on + `0.0.0.0:8889` with `resource_to_telemetry_conversion.enabled: true` so the + tier and instance resource attributes become Prometheus labels. +- **Extensions.** `health_check` on `0.0.0.0:13133` only. There is **no** + `zpages` extension. + +Deliberately absent from the base config — do not document them as present: +no `memory_limiter`, no `tail_sampling`, no Elastic APM exporter, and no +`tx_account` attribute rule (the hashed keys are the two `pathfind_*_account` +ones). ### 5.5.2 Production Configuration -The production collector adds TLS on the OTLP gRPC receiver and a richer processor chain: a `memory_limiter` (OOM guard), `batch` (5s timeout, size 512), `tail_sampling`, and an `attributes` processor that hashes sensitive fields (e.g. `tx_account`) and stamps `deployment.environment`. Tail sampling keeps all `ERROR` traces, slow consensus rounds (>5s) and slow RPC requests (>1s), and probabilistically samples the remainder at 10%. Exporters target Grafana Tempo (TLS) and Elastic APM; `health_check` and `zpages` extensions are enabled for operability. +There is no separate "production" collector config in this repo. The one +overlay that exists is `docker/telemetry/otel-collector-config.grafanacloud.yaml`. +It is **not** the base config plus one processor — it restructures the service +graph. The full delta: + +| Added by the overlay | Where | Purpose | +| ------------------------ | ------ | ------------------------------------------------------------------------- | +| `basicauth/grafanacloud` | `:29` | Extension; instance id / API token from the container environment | +| `tail_sampling` | `:60` | One `probabilistic` policy at **0.5%**, `decision_wait: 10s` | +| `transform/cloudlabels` | `:119` | Copies three resource attrs onto datapoint labels for Cloud (OTLP) ingest | +| `otlphttp/grafanacloud` | `:236` | Single OTLP/HTTP exporter fanning all three signals to Grafana Cloud | +| `metrics_flush_interval` | `:136` | `spanmetrics` flushes every 15s instead of the 60s default | + +| Removed by the overlay | Consequence | +| ---------------------- | ---------------------------------------------------------------------------- | +| `attributes/hash` | **Pathfinding account attributes are not hashed on this config** — see below | +| `debug` | No console span dump; collector logs alone when diagnosing ingest | + +Pipelines go from **three** (`traces`, `metrics`, `logs`) to **five** +(`:253-280`): `traces/metrics`, `traces/store`, `metrics/local`, +`metrics/cloud`, `logs`. `tail_sampling` is applied in **`traces/store`** +(`:259-261`) — the branch feeding Tempo and Grafana Cloud — not in a pipeline +named `traces`, which does not exist in the overlay. The `traces/metrics` +branch feeds `spanmetrics` unsampled, so the derived RED metrics stay exact +while stored traces are ~1/200 of ingested ones. + +> **Known issue — the cloud path does not hash pathfinding accounts.** The base +> config runs `attributes/hash` on its `traces` pipeline +> (`otel-collector-config.yaml:105-110`), hashing `pathfind_source_account` and +> `pathfind_dest_account` as defense in depth behind the node-side hashing. The +> overlay declares no such processor and lists none on any of its five +> pipelines, so on the Grafana Cloud config those two attributes reach **both** +> Grafana Cloud and the local Tempo with whatever value the node sent. Any node +> that emits raw addresses loses its second line of defense. Adding +> `attributes/hash` to `traces/store` and `traces/metrics` would close the gap. + +Hardening a collector for a real deployment (TLS/mTLS on the receiver, +NetworkPolicy, peer trace-context validation) is covered in +[Securing the OTel Pipeline](./secure-OTel.md) — not by any config file in +`docker/telemetry/`. --- @@ -119,7 +354,31 @@ The production collector adds TLS on the OTLP gRPC receiver and a richer process > **OTLP** = OpenTelemetry Protocol -The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up four services on a shared `xrpld-telemetry` network: an `otel-collector` (otel/opentelemetry-collector-contrib) exposing OTLP gRPC `4317`, OTLP HTTP `4318`, and health check `13133`; `tempo` for trace storage/visualization; `grafana` with provisioned datasources and dashboards (anonymous admin enabled); and an optional `prometheus` for metric correlation. +The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up **six** services on a shared `xrpld-telemetry` bridge network. All images are pinned to exact tags. + +| Service | Image | Published ports | Role | +| ---------------- | ---------------------------------------------- | ---------------------- | ---------------------------------------------------------------- | +| `otel-collector` | `otel/opentelemetry-collector-contrib:0.158.0` | `4317`, `4318`, `8889` | OTLP ingest, spanmetrics, filelog tail, Prometheus scrape target | +| `tempo` | `grafana/tempo:2.9.4` | `3200` | Trace storage and TraceQL | +| `loki` | `grafana/loki:3.7.6` | `3100` | Log storage for log↔trace correlation | +| `prometheus` | `prom/prometheus:v3.13.2` | `9090` | Scrapes the collector's `:8889` | +| `grafana` | `grafana/grafana:13.1.2` | `3000` | Dashboards + provisioned datasources/alerts, anonymous admin | +| `renderer` | `grafana/grafana-image-renderer:v5.12.0` | `8081` | Panel→PNG rendering for image export and alert screenshots | + +Two corrections to earlier drafts: + +- **`prometheus` is not optional.** `grafana` lists it in `depends_on` (along + with `tempo`, `loki` and `renderer`), and 7 of the 15 dashboards query + `span_calls_total` from it. Removing it blanks most panels. +- **Port `13133` is not published.** The collector's `health_check` extension + listens on `13133` inside the container, but the base compose file publishes + only `4317`, `4318` and `8889`. Health checks from the host must either add a + port mapping or run `docker compose exec`. + +The collector also bind-mounts the xrpld log root read-only +(`${XRPLD_LOG_DIR:-./data/logs}` → `/var/log/xrpld`) for the `filelog` +receiver, and the `grafana` service reads Slack/email alert secrets from an +optional gitignored `.env.alerting`. --- @@ -131,7 +390,7 @@ The authoritative development stack lives in the repo at `docker/telemetry/docke flowchart TB subgraph config["Configuration Sources"] cfgFile["xrpld.cfg
[telemetry] section"] - cmake["CMake
XRPL_ENABLE_TELEMETRY"] + cmake["CMake option: telemetry
ON today for CI, OFF once merged
when ON, defines XRPL_ENABLE_TELEMETRY"] end subgraph init["Initialization"] @@ -168,7 +427,7 @@ flowchart TB **Reading the diagram:** -- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake flag controls whether telemetry is compiled in at all. Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector. +- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake `telemetry` option controls whether telemetry is compiled in at all. That option is declared ON today only so CI compiles the instrumented paths; OFF is the intended default once merged, so treat the build gate as something to pass explicitly, and the runtime gate is opt-in either way (`enabled=0` by default). Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector. - **Initialization**: `makeTelemetrySetup()` parses config values, then `makeTelemetry()` constructs the provider, processor, and exporter objects. - **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire. - **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP/HTTP and enters the external Collector pipeline. (OTLP/gRPC is future work — see design decisions §2.2.2.) @@ -184,29 +443,88 @@ Step-by-step instructions for integrating xrpld traces with Grafana. ### 5.8.1 Data Source Configuration -#### Tempo (Recommended) +Three datasources are provisioned from `docker/telemetry/grafana/provisioning/datasources/`. There is **no** Elastic APM datasource — `elastic-apm.yaml` was described in an earlier draft but never existed. Elastic remains a _possible_ backend (§7.2); nothing in this repo provisions it. -A Tempo datasource (`grafana/provisioning/datasources/tempo.yaml`, provisioned from `docker/telemetry/grafana/`) points at `http://tempo:3200` and enables `tracesToLogs` (linking to Loki on `service.name`/`tx_hash` and mapping `trace_id` → `traceID`), `serviceMap` against Prometheus, the node graph, and Loki search. +| File | Type | URL | uid | Notes | +| ----------------- | ------------ | ------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------- | +| `tempo.yaml` | `tempo` | `http://tempo:3200` | `tempo` | `nodeGraph`, `serviceMap`/`tracesToMetrics` → `prometheus`, `tracesToLogs` → `loki`, plus ~30 Explore search filters | +| `prometheus.yaml` | `prometheus` | `http://prometheus:9090` | `prometheus` | Backs every span-metric and native-metric panel | +| `loki.yaml` | `loki` | `http://loki:3100` | `loki` | Backs `log-derived-insights`; derived fields jump back to Tempo | -#### Elastic APM +The Tempo `tracesToLogs` block is configured as `filterByTraceID: true`, +`filterBySpanID: false`, **`tags: []`**. The empty tag list is deliberate: the +correlation is by trace ID alone, so no span attribute needs to exist on both +sides. Earlier drafts claimed `trace_id` + `tx_hash` tags — that is not what +ships, and adding a tag Tempo cannot resolve blanks the link. -Alternatively, an Elasticsearch datasource (`grafana/provisioning/datasources/elastic-apm.yaml`) of type `elasticsearch` points at `http://elasticsearch:9200` against the `apm-*` index, using `@timestamp` as the time field and mapping the log message/level fields. +The search-filter list is the practical index of queryable span attributes: +resource scope (`service.name`, `service.instance.id`, `service.version`, +`xrpl.network.id`, `xrpl.network.type`), intrinsics (`name`, `status`, +`duration`), and span scope (`command`, `rpc_status`, `rpc_role`, `tx_hash`, +`tx_type`, `tx_status`, `local`, `path`, `suppressed`, `peer_version`, +`consensus_*`, `ledger_seq`, `ledger_hash`, `close_time_correct`, +`close_resolution_ms`, `proposers`, `mode_old`, `mode_new`, `txq_status`, +`ter_code`). ### 5.8.2 Dashboard Provisioning -A dashboard provider (`grafana/provisioning/dashboards/dashboards.yaml`) loads the `xrpld` dashboard folder from disk (`/var/lib/grafana/dashboards/rippled`), polling for changes every 30s with deletion disabled. +`grafana/provisioning/dashboards/dashboards.yaml` declares a single `file` +provider named `xrpld-telemetry`, `orgId: 1`, targeting Grafana folder `xrpld` +from path `/var/lib/grafana/dashboards` (no `/rippled` suffix), with +`disableDeletion: false`, `editable: true`, `foldersFromFilesStructure: false`. +It sets **no** poll interval — Grafana's `updateIntervalSeconds` default +applies; the "every 30s" figure in earlier drafts was invented. -### 5.8.3 Example Dashboard: RPC Performance +`docker-compose.yml` mounts `./grafana/dashboards` read-only at that path, so +the 15 JSON files in `docker/telemetry/grafana/dashboards/` are what gets +provisioned. -An example `xrpld RPC Performance` dashboard (uid `xrpld-rpc-performance`) sourced from Tempo via TraceQL provides four panels: RPC latency by command (heatmap), RPC error rate by command (timeseries), the top 10 slowest RPC commands by average duration (table), and a recent-traces table. +### 5.8.3 Shipped Dashboards -### 5.8.4 Example Dashboard: Transaction Tracing +The dashboards are Prometheus-first, not TraceQL-first, and their uids are +bare (no `xrpld-` prefix). The full inventory and per-panel query reference is +[09-data-collection-reference.md](./09-data-collection-reference.md); the uids +are: -An example `xrpld Transaction Tracing` dashboard (uid `xrpld-tx-tracing`) over Tempo provides three panels: transaction throughput (`tx.receive` rate, stat), cross-node relay count (average `span.relay_count` on `tx.relay`, timeseries), and a table of transaction validation errors (`tx.validate` with `status.code=error`). +`consensus-health`, `fee-market`, `job-queue`, `ledger-data-sync`, +`ledger-operations`, `log-derived-insights`, `network-traffic`, `node-health`, +`overlay-traffic-detail`, `peer-network`, `peer-quality`, `rpc-pathfinding`, +`rpc-performance`, `transaction-overview`, `validator-health`. -### 5.8.5 TraceQL Query Examples +> **Panel-count convention used in these docs**: counts are of **data panels +> only** — `type: "row"` collapsible headers are excluded, because a row is a +> layout element with no query. A board's raw `panels` array is therefore longer +> than its stated count (e.g. `rpc-performance` has 19 array entries: 2 rows + +> 17 data panels). -Common queries for xrpld traces: +Two examples described in earlier drafts do not exist and should not be looked +for: `xrpld-rpc-performance` (the real board is `rpc-performance`, **17** data +panels in 2 rows, mostly Prometheus span metrics) and `xrpld-tx-tracing` (the +transaction board is `transaction-overview`, **18** data panels in 3 rows; its +error panel filters `span_calls_total{span_name="tx.process", +ter_result!~"tesSUCCESS|"}`, since no `tx.validate` span was ever built — see +[02 §2.3.2](./02-design-decisions.md)). + +> **Why `!~"tesSUCCESS|"` and not `!="tesSUCCESS"`.** An absent Prometheus label +> compares equal to the empty string, and `tx.process` can end **without** a +> `ter_result` attribute: `processTransaction()` returns early when +> `preProcessTransaction()` rejects the transaction +> (`NetworkOPs.cpp:1437-1438`) and `doTransactionAsync()` returns early when the +> transaction is already applying (`:1461-1462`); the only setter runs later, at +> `:1674`. Those series carry `ter_result=""`, which `!="tesSUCCESS"` counts as +> an error. The regex form excludes the empty value explicitly (the trailing +> `|` alternative), which is the form `docs/telemetry-runbook.md:1198` and two +> of the three `transaction-overview.json` failure panels already use. + +Every dashboard exposes a `$node` template variable bound to +`service_instance_id`; see the §5.1.1 note on why `service_instance_id` must be +set for metric panels to split per node. + +### 5.8.4 TraceQL Query Examples + +Common queries for xrpld traces. Every span name and attribute below is one +that the code actually emits — check against the `*SpanNames.h` constants +before adding more. ``` # Find all traces for a specific transaction hash @@ -218,57 +536,163 @@ Common queries for xrpld traces: # Find consensus rounds taking >5 seconds {resource.service.name="xrpld" && name="consensus.round"} | duration > 5s -# Find failed transactions with error details -{resource.service.name="xrpld" && name="tx.validate" && status.code=error} +# Find failed transaction processing +{resource.service.name="xrpld" && name="tx.process" && span.ter_result!="tesSUCCESS"} -# Find transactions relayed to many peers -{resource.service.name="xrpld" && name="tx.relay"} | span.relay_count > 10 +# Find failed apply-pipeline stages (preflight / preclaim / transactor) +{resource.service.name="xrpld" && name=~"tx\\.(preflight|preclaim|transactor)" && status=error} + +# Find transactions that arrived from a peer rather than a local client. +# The `local` attribute lives on tx.process, NOT on tx.receive (see the note +# below). +{resource.service.name="xrpld" && name="tx.process" && span.local=false} # Compare latency across nodes {resource.service.name="xrpld" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id) ``` -### 5.8.6 Correlation with PerfLog +> Queries in earlier drafts used `tx.validate`, `tx.relay` and +> `span.relay_count`. None of the three exists: signature/format validation +> ships as `tx.preflight`/`tx.preclaim`, and no relay span or relay-count +> attribute was ever built. See [02 §2.3.2](./02-design-decisions.md). -To correlate OpenTelemetry traces with existing PerfLog data: +> **TraceQL silently returns nothing for an absent attribute.** Unlike PromQL, +> where a missing label compares equal to `""`, a TraceQL attribute predicate +> matches only spans that actually carry the attribute — including negated +> forms such as `!=` and `=~".*"`. So filtering on the wrong span name yields +> zero rows with no error. `local` has exactly one set-site, +> `NetworkOPs.cpp:1417`, and it is on **`tx.process`**: an earlier draft paired +> it with `name="tx.receive"`, which can never match. Check the attribute's +> owning span in +> [09 §1.2](./09-data-collection-reference.md) before combining a `name=` and a +> `span.` predicate. -**Step 1: Configure Loki to ingest PerfLog** +### 5.8.5 Correlation with Logs -Configure a Promtail scrape job (`promtail-config.yaml`) that tails `/var/log/rippled/perf*.log`, parses each JSON line, and promotes `trace_id`, `ledger_seq`, and `tx_hash` to Loki labels. +Log↔trace correlation is **implemented** (Phase 8) and needs no Promtail, +Fluentd or PerfLog change. Two pieces: -**Step 2: Add trace_id to PerfLog entries** +1. **The node stamps the IDs.** The journal sink `Logs::format()` + (`src/libxrpl/basics/Log.cpp:304-338`, guarded by `XRPL_ENABLE_TELEMETRY`) + reads the thread-local OTel context and, when a valid span is active, + prefixes the message with `trace_id=<32 hex> span_id=<16 hex>`. It reads + the context value directly rather than calling `GetSpan()` to avoid a heap + allocation on the (common) no-span path. This is the ordinary `debug.log` + stream — PerfLog is not involved, and the `setTraceId` hook described in + earlier drafts was never built. +2. **The collector ingests them.** The `filelog` receiver tails + `/var/log/xrpld/*/debug.log` and its `regex_parser` lifts `trace_id` and + `span_id` as optional capture groups (§5.5.1). `resource/logs` applies an + `upsert` of `service.name=xrpld`, which Loki promotes to the stream label + `service_name`, so the canonical selector is **`{service_name="xrpld"}`**. + Logs land in Loki via `otlphttp/loki`. -Modify PerfLog so its JSON output includes a `trace_id` field whenever a valid span is active: fetch the current span from the OpenTelemetry runtime context, and if its context is valid, render the trace ID as a 32-character lowercase hex string into the log entry. +> **Known issue — the collector's `job` upsert is ineffective for stream +> selection.** `resource/logs` also applies an `upsert` of a `job=xrpld` attribute +> (`otel-collector-config.yaml:62-70`) with the stated intent that operators +> could paste `{job="xrpld"}`. That does not work. On OTLP ingest Loki promotes +> only an **allow-listed** set of resource attributes to indexed stream labels +> (`service.name`, `service.namespace`, `service.instance.id`, +> `deployment.environment`, the `k8s.*`/`cloud.*` keys); `job` is not on that +> list, and this repo ships no Loki config override — `docker-compose.yml:75` +> starts Loki with the image's built-in `/etc/loki/local-config.yaml`. `job` +> therefore lands in **structured metadata**, which cannot appear in a stream +> selector, so `{job="xrpld"}` returns an empty result rather than an error. +> Corroboration in-repo: `docs/telemetry-runbook.md:2533` states the same +> ("`service_name="xrpld"` (not `job="xrpld"`)"), and **all 38 Loki queries** in +> the shipped dashboards (35 panel targets + 3 Loki-backed template variables) +> select on `service_name` — **zero** use `job`. Either drop the `job` +> upsert or add `job` to Loki's `distributor.otlp_config.resource_attributes` +> allow-list via a mounted Loki config; until then, use `service_name`. -**Step 3: Configure Grafana trace-to-logs link** +Grafana then links the two directions: the Tempo datasource's `tracesToLogs` +(`filterByTraceID: true`, `tags: []`) jumps trace → logs, and `loki.yaml`'s +derived fields jump log → trace. -In the Tempo datasource, set the `tracesToLogs` derived field to link to Loki on the `trace_id` and `tx_hash` tags, with `filterByTraceID: true`. - -### 5.8.7 Correlation with Insight/OTel System Metrics +### 5.8.6 Correlation with Insight/OTel System Metrics To correlate traces with Beast Insight system metrics: **Step 1: Export Insight metrics to Prometheus** Beast Insight metrics are exported natively via OTLP to the OTel Collector, -which exposes them on the Prometheus endpoint alongside spanmetrics. Configure -the `[insight]` section of `xrpld.cfg` with `server=otel`, -`endpoint=http://localhost:4318/v1/metrics`, and `prefix=xrpld`; no separate -StatsD exporter or Prometheus scrape job is needed when using `server=otel`. +which exposes them on its Prometheus endpoint (`:8889`) alongside spanmetrics. +Set `server=otel` in the `[insight]` section of `xrpld.cfg`; no separate StatsD +exporter or Prometheus scrape job is needed. -**Step 2: Add exemplars to metrics** +`makeCollectorManager()` (`src/xrpld/app/main/CollectorManager.cpp`) reads these +`[insight]` keys: -The OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter, linking metric spikes to specific traces. +| Key | Read at | Effect when `server=otel` | +| --------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| `server` | `:35` | **Live.** `statsd` \| `otel` \| anything else. Selects the collector implementation. | +| `address` | `:39` | StatsD only — the UDP endpoint. | +| `prefix` | `:41`, `:53` | **Inert.** Stored on the OTel collector but `formatName()` prepends nothing (`OTelCollector.cpp:855-866`); only StatsD applies it. | +| `endpoint` | `:50` | **Inert.** Logged for diagnostics (`OTelCollector.cpp:730`), then unused. | +| `service_instance_id` | `:58` | **Inert.** `(void)`-discarded (`OTelCollector.cpp:722`). | +| `service_name` | `:64` | **Inert.** `(void)`-discarded (`OTelCollector.cpp:723`). | -**Step 3: Configure Grafana metric-to-trace link** +> **Where the identity and endpoint actually come from.** `OTelCollector` +> deliberately does **not** own a pipeline: it fetches the Meter from the +> **global** MeterProvider that `Telemetry::initMetrics()` published +> (`OTelCollector.cpp:726-745`). So the resource attributes — including +> `service.instance.id`, which every dashboard filters on — and the exporter +> URL both come from the **`[telemetry]`** section, not `[insight]`. The four +> inert keys above are back-compat leftovers from the StatsD-era signature; +> setting them has no effect. Set `[telemetry] service_instance_id` instead +> (§5.1.1). -In the Prometheus datasource, set `exemplarTraceIdDestinations` to map the `trace_id` exemplar to the Tempo datasource. +> **`server=otel` is not the default.** `CollectorManager.cpp:72-75` falls through +> to `NullCollector` for any unrecognised or absent `server` value, so a node +> with no `[insight]` section emits no metrics at all. -**Step 4: Dashboard panel with exemplars** +**Step 2: Correlate metrics to traces** -Add a timeseries panel over Prometheus (e.g. `histogram_quantile(0.99, rate(rpc_duration_seconds_bucket[5m]))`) with `exemplar: true` enabled. +Today this is a **time-range** correlation, not a click-through one: note the +window from the metric panel, then search Tempo over the same window filtered +by `service.instance.id`. -This allows clicking on metric data points to jump directly to the related trace. +> **Exemplars are NOT implemented.** Earlier drafts of this section instructed +> operators to rely on automatic exemplars, set +> `exemplarTraceIdDestinations` on the Prometheus datasource, and enable +> `exemplar: true` on panels. None of that is wired up: the string `exemplar` +> appears **nowhere** in `src/libxrpl/telemetry/`, `src/xrpld/telemetry/`, or +> `docker/telemetry/`. Concretely, three things are missing — +> +> 1. the SDK's exemplar filter is left at its default and no reservoir is +> configured in `Telemetry::initMetrics()` or `MetricsRegistry`; +> 2. the collector's `prometheus` exporter has no exemplar settings; +> 3. `grafana/provisioning/datasources/prometheus.yaml` has no +> `exemplarTraceIdDestinations` block. +> +> Note also that the query used as an example, `rpc_duration_seconds_bucket`, +> does not exist — RPC latency histograms are `span_duration_milliseconds_bucket` +> (spanmetrics, `unit: ms`) and `rpc_method_us` (native). Wiring exemplars end +> to end is genuine open work; until it lands, do not document a click-through +> that operators cannot perform. + +**Step 3: Jump the other way instead** + +Trace → metrics is available now: the Tempo datasource sets +`tracesToMetrics.datasourceUid: prometheus` with a ±1h time shift, so the +span-metric queries it builds resolve against the `span_*` families the +collector's `spanmetrics` connector produces. Trace → logs and log → trace are +both live (§5.8.5). + +> **Known gap — Service Map is configured but inactive.** The Tempo datasource +> declares `serviceMap.datasourceUid: prometheus`, and `tempo.yaml:70-76` +> enables the `service-graphs` metrics-generator processor, but the generator +> has nowhere to write: its `remote_write` block is **commented out** +> (`tempo.yaml:53-56`), and `prometheus.yml:6-9` defines a single scrape job +> against `otel-collector:8889` — it never scrapes or accepts writes from +> Tempo. `traces_service_graph_request_total` and its siblings are therefore +> never stored, so the Service Map / Node Graph tab renders empty. The same gap +> means Tempo's `span-metrics` processor never lands +> `traces_spanmetrics_*` either (§5.5.1) — every span metric the dashboards use +> comes from the collector's connector instead. Closing it needs both halves: +> uncomment `remote_write` in `tempo.yaml` **and** enable +> `--web.enable-remote-write-receiver` on the Prometheus service (or add a +> scrape job for Tempo). --- diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 55f833102d..4e925d16fb 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -62,7 +62,7 @@ gantt section Phase 8 Log-Trace Correlation :p8, after p7, 1w - section Phase 9 (Future) + section Phase 9 Internal Metric Gap Fill :p9, after p8, 2.5w section Phase 10 (Future) @@ -93,11 +93,18 @@ gantt ### Exit Criteria -- [ ] OpenTelemetry SDK compiles and links -- [ ] Telemetry can be enabled/disabled via config -- [ ] Basic span creation works -- [ ] No performance regression when disabled -- [ ] Unit tests passing +- [x] OpenTelemetry SDK compiles and links — `conanfile.py:153` requires + `opentelemetry-cpp/1.28.0` when the `telemetry` option is on (`:152`); + `cmake/XrplCore.cmake:91,245` links the umbrella target + `opentelemetry-cpp::opentelemetry-cpp` +- [x] Telemetry can be enabled/disabled via config — `TelemetryConfig.cpp:103` + parses `[telemetry] enabled` (default 0) +- [x] Basic span creation works — `libxrpl/telemetry/SpanGuard.cpp`, covered by + `src/tests/libxrpl/telemetry/SpanGuardScope.cpp` and `SpanGuardFactory.cpp` +- [ ] No performance regression when disabled — `NullTelemetry.cpp` provides the + no-op path, but the <0.1% claim needs the Phase 10 benchmark suite + (`--with-benchmark`), which is not run in CI +- [x] Unit tests passing — 10 GTest files under `src/tests/libxrpl/telemetry/` --- @@ -124,11 +131,20 @@ gantt ### Exit Criteria -- [ ] All RPC commands traced -- [ ] Trace context propagates from HTTP headers -- [ ] WebSocket and HTTP both instrumented -- [ ] <1ms overhead per RPC call -- [ ] Integration tests passing +- [x] All RPC commands traced — `rpc.command.{name}` built from + `rpc_span::prefix::command` (`RpcSpanNames.h:127`), emitted from + `RPCHandler.cpp` +- [ ] Trace context propagates from HTTP headers — **not implemented**. + `TraceContextPropagator.h` only offers `extractFromProtobuf()` / + `injectToProtobuf()`; there is no `traceparent` header reader anywhere in + the tree (`grep -ri traceparent src/ include/` → 0 hits). Cross-node + correlation is carried by the protobuf `TraceContext` field and by + deterministic trace IDs instead. +- [x] WebSocket and HTTP both instrumented — `rpc.http_request` and + `rpc.ws_message` (`RpcSpanNames.h:133-136`) +- [ ] <1ms overhead per RPC call — needs the Phase 10 benchmark suite +- [ ] Integration tests passing — the end-to-end RPC span assertions live in the + Phase 10 harness (`validate_telemetry.py`), not on this branch --- @@ -162,13 +178,19 @@ and [Phase3_taskList.md Task 3.9](./Phase3_taskList.md) for the full implementat ### Exit Criteria -- [ ] Transaction traces span across nodes -- [ ] Trace context in Protocol Buffer messages -- [ ] HashRouter deduplication visible in traces -- [ ] Multi-node integration tests passing -- [ ] <5% overhead on transaction throughput -- [ ] Deterministic trace_id: all nodes produce same trace_id for same transaction -- [ ] Protobuf span_id propagation preserves parent-child ordering when available +- [ ] Transaction traces span across nodes — needs a live multi-node run (Phase 10 harness) +- [x] Trace context in Protocol Buffer messages — `message TraceContext` + (`include/xrpl/proto/xrpl.proto:101`), carried as optional field `1001` on + three message types (`:130`, `:181`, `:229`) +- [x] HashRouter deduplication visible in traces — `suppressed` attribute + (`TxSpanNames.h:71`) +- [ ] Multi-node integration tests passing — Phase 10 harness +- [ ] <5% overhead on transaction throughput — needs the Phase 10 benchmark suite +- [x] Deterministic trace_id: all nodes produce same trace_id for same transaction + — `libxrpl/telemetry/DeterministicIdGenerator.cpp` +- [x] Protobuf span_id propagation preserves parent-child ordering when available + — `TraceContextPropagator.h` `injectToProtobuf()` / `extractFromProtobuf()` + (`trace_state`, field 4, is reserved and deliberately unwired) --- @@ -187,7 +209,7 @@ and [Phase3_taskList.md Task 3.9](./Phase3_taskList.md) for the full implementat | 4.5 | Add consensus-specific attributes | ✅ Done | | 4.6 | Correlate with transaction traces | ✅ Done | | 4.7 | Build verification and testing | ✅ Done | -| 4.8 | Validation span enrichment (ext. dashboard) | ❌ Not done | +| 4.8 | Validation span enrichment (ext. dashboard) | ✅ Done (partial) | **Note**: The original plan doc listed tasks 4.7-4.11 as "Validator list tracing", "Amendment voting tracing", "SHAMap sync tracing", "Multi-validator integration tests", @@ -212,10 +234,18 @@ SHAMap tracing are not implemented. - [x] Phase transitions visible (open, establish, close, accept) - [x] Proposals and validations traced — send and receive; relay deferred to Phase 4b - [x] Close time agreement tracked (per `avCT_CONSENSUS_PCT`) -- [x] No impact on consensus timing -- [ ] Multi-validator test network validated +- [ ] No impact on consensus timing — **not measured**. No consensus-timing + benchmark has been run on any branch in the chain; the benchmark suite + lives on the Phase 10 branch and does not isolate consensus round time +- [ ] Multi-validator test network validated — needs a live multi-node run; the + 5-node harness lives on the Phase 10 branch, not here - [x] Transaction-consensus correlation (Task 4.6) — `tx.included` events in doAccept -- [ ] Validation span enrichment (Task 4.8) — not implemented +- [x] Validation span enrichment (Task 4.8) — send span sets `ledger_seq`, + `ledger_hash`, `proposing`, `full_validation` (`RCLConsensus.cpp:975-981`); + receive span sets `ledger_hash`, `full_validation` (`PeerImp.cpp:2573-2574`); + `consensus.accept` sets `quorum` from `app_.getValidators().quorum()` + (`RCLConsensus.cpp:516`). Still open: `proposers_validated` — never + implemented, no attribute of that name exists in the tree. ### Implementation Status — Phase 4a Complete @@ -285,7 +315,7 @@ with `TraceCategory::Consensus` gating. No macros used — all tracing via direc - [x] Strategy switchable via config (`deterministic` / `attribute`) - [x] Consecutive rounds linked via follows-from spans - [x] Build passes with telemetry ON and OFF -- [x] No impact on consensus timing +- [ ] No impact on consensus timing — **not measured** (see §6.5 Exit Criteria) See [Phase4_taskList.md](./Phase4_taskList.md) for full task details. @@ -376,23 +406,35 @@ The `StatsDMeterImpl` in `StatsDCollector.cpp` sends metrics with `|m` suffix, w ### New Grafana Dashboards -**Node Health** (`statsd-node-health.json`, uid: `xrpld-statsd-node-health`): +**Node Health** (`node-health.json`, uid: `node-health`): - Validated/Published Ledger Age, Operating Mode Duration/Transitions, I/O Latency, Job Queue Depth, Ledger Fetch Rate, Ledger History Mismatches, Key Jobs Execution/Dequeue Time, FullBelowCache Size/Hit Rate, Ledger Publish Gap, State Duration Rate, All Jobs Detail -**Network Traffic** (`statsd-network-traffic.json`, uid: `xrpld-statsd-network`): +**Network Traffic** (`network-traffic.json`, uid: `network-traffic`): - Active Inbound/Outbound Peers, Peer Disconnects, Total Bytes/Messages In/Out, Transaction/Proposal/Validation Traffic, Top Traffic Categories, Duplicate Traffic, All Traffic Categories Detail -**RPC & Pathfinding (StatsD)** (`statsd-rpc-pathfinding.json`, uid: `xrpld-statsd-rpc`): +**RPC & Pathfinding** (`rpc-pathfinding.json`, uid: `rpc-pathfinding`): - RPC Request Rate, Response Time p95/p50, Response Size p95/p50, Pathfinding Fast/Full Duration, Resource Warnings/Drops, Response Time Heatmap ### Exit Criteria -- [ ] StatsD metrics visible in Prometheus (`curl localhost:9090/api/v1/query?query=ledgermaster_validated_ledger_age`) -- [ ] All 3 new Grafana dashboards load without errors +- [x] StatsD metrics visible in Prometheus (`curl localhost:9090/api/v1/query?query=ledgermaster_validated_ledger_age`) + — superseded by Phase 7: the same metric names now arrive over OTLP + (`server=otel`) and the StatsD receiver has been removed from the collector +- [x] All 3 new Grafana dashboards load without errors — shipped as + `node-health.json`, `network-traffic.json`, `rpc-pathfinding.json`, + uids `node-health` / `network-traffic` / `rpc-pathfinding`. These three + were renamed in **two** steps: `statsd-*.json` → `system-*.json` + (`2f7064ace6`), then `system-*.json` → bare (`2c590a47c5`). An + `xrpld-statsd-*` form **never existed** in any commit, and `25868f2740` + did not touch these three — it de-prefixed a different set + (`xrpld-fee-market`, `xrpld-job-queue`, `xrpld-peer-quality`, + `xrpld-validator-health` → bare). §6.7 above now carries the shipped names. - [ ] Integration test verifies at least core StatsD metrics (ledger age, peer counts, RPC requests) + — the metric assertions live in the Phase 10 harness + (`expected_metrics.json`), not on this branch - [ ] ~~Meter metrics (`warn`, `drop`) flow correctly after `|m` → `|c` fix~~ — DEFERRED (breaking change, tracked separately; resolved by Phase 7's OTel Counter mapping) --- @@ -568,20 +610,28 @@ See [Phase7_taskList.md](./Phase7_taskList.md) for detailed per-task breakdown. ### Exit Criteria - [ ] All 255+ metrics visible in Prometheus via OTLP pipeline (no StatsD receiver) -- [ ] `server=otel` is the default in development docker-compose -- [ ] `server=statsd` still works as a fallback -- [ ] Existing Grafana dashboards display data correctly -- [ ] Integration test passes with OTLP-only metrics pipeline -- [ ] No performance regression vs StatsD baseline (< 1% CPU overhead) -- [ ] Deferred Task 6.1 (`|m` wire format) no longer relevant + — the receiver is gone and `OTelCollector` is wired, but the 255+ figure + needs a live scrape to confirm +- [x] `server=otel` is the default in development docker-compose — + `docker/telemetry/xrpld-telemetry.cfg:112`, + `xrpld-telemetry-mainnet.cfg:121`, `integration-test.sh:380` +- [x] `server=statsd` still works as a fallback — `CollectorManager.cpp:37` + still branches on `server == "statsd"` alongside `"otel"` (`:46`) +- [ ] Existing Grafana dashboards display data correctly — needs a live stack +- [ ] Integration test passes with OTLP-only metrics pipeline — Phase 10 harness +- [ ] No performance regression vs StatsD baseline (< 1% CPU overhead) — needs + the Phase 10 benchmark suite +- [x] Deferred Task 6.1 (`|m` wire format) no longer relevant — `OTelMeterImpl` + (`OTelCollector.cpp:308`) maps meters onto an OTel counter, so the + non-standard `|m` wire type is never emitted on the `server=otel` path --- -## 6.9 Phase 8: Log-Trace Correlation and Centralized Log Ingestion (Week 13) +## 6.8.1 Phase 8: Log-Trace Correlation and Centralized Log Ingestion (Week 13) ### Motivation -xrpld's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. +xrpld's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active, sampled span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. #### Gains @@ -599,7 +649,7 @@ xrpld's `beast::Journal` logs and OpenTelemetry traces are currently two disjoin #### Decision -The correlation value far outweighs the risks. The log format change is backward-compatible (fields are appended only when a span is active), and the filelog receiver regex is straightforward to maintain. +The correlation value far outweighs the risks. The log format change is backward-compatible (fields are appended only when a sampled span is active), and the filelog receiver regex is straightforward to maintain. ### Architecture @@ -675,19 +725,37 @@ flowchart LR ### Exit Criteria -- [ ] Log lines within active spans contain `trace_id= span_id=` -- [ ] Log lines outside spans have no trace context (no empty fields) -- [ ] Loki ingests xrpld logs via OTel Collector filelog receiver -- [ ] Grafana Tempo → Loki one-click correlation works -- [ ] Grafana Loki → Tempo reverse lookup works via derived field -- [ ] Integration test verifies trace_id presence in logs -- [ ] No performance regression from trace_id injection (< 0.1% overhead) +- [x] Log lines within active spans contain `trace_id= span_id=` — + `Log.cpp:304-338`, guarded by `#ifdef XRPL_ENABLE_TELEMETRY` +- [x] Log lines outside spans have no trace context (no empty fields) — the + block reads the thread-local span key and appends nothing when it is + absent or the context is invalid (`Log.cpp:310-318`) +- [x] Loki ingests xrpld logs via OTel Collector filelog receiver — + `otel-collector-config.yaml:38` (`filelog`); `loki` service in + `docker-compose.yml:71` +- [x] Grafana Tempo → Loki one-click correlation works — + `provisioning/datasources/tempo.yaml:32` (`tracesToLogs`) +- [x] Grafana Loki → Tempo reverse lookup works via derived field — + `provisioning/datasources/loki.yaml:16` (`derivedFields`) +- [ ] Integration test verifies trace_id presence in logs — implemented in the + Phase 10 harness, but CI runs it with `--skip-loki`, so it is not gated +- [ ] No performance regression from trace_id injection (< 0.1% overhead) — + needs the Phase 10 benchmark suite --- -## 6.8.2 Phase 9: Internal Metric Instrumentation Gap Fill (Weeks 14-15) — Future Enhancement +## 6.8.2 Phase 9: Internal Metric Instrumentation Gap Fill (Weeks 14-15) -> **Status**: Planned, not yet implemented. +> **Status**: Complete. Merged on `pratik/otel-phase9-metric-gap-fill`. Shipped +> artefacts: `src/xrpld/telemetry/MetricsRegistry.{h,cpp}` (~41 KB + ~71 KB), +> `src/xrpld/telemetry/MetricMacros.h`, `include/xrpl/nodestore/WriteStats.h`, +> `src/xrpld/app/ledger/AcquireStats.h`, +> `include/xrpl/telemetry/GetObjectMetricNames.h`, 10 GTest files under +> `src/tests/libxrpl/telemetry/`, 4 new Grafana dashboards, provisioned Grafana +> alerting (13 rules), and the Phase 9 sections of +> `09-data-collection-reference.md` and `docs/telemetry-runbook.md`. +> Tasks 9.14-9.17 remain open by design — see +> [Phase9_taskList.md](./Phase9_taskList.md). ### Motivation @@ -700,50 +768,64 @@ Hybrid approach — two instrumentation strategies based on proximity to existin ```mermaid flowchart TB subgraph xrpld["xrpld process"] - subgraph existing["Existing beast::insight registrations"] - NS["NodeStore I/O
(Database.cpp)"] + subgraph newreg["New OTel MetricsRegistry (all Phase 9 metrics)"] + NS["NodeStore I/O
async gauge
nodestore_state"] + CR["Cache Hit Rates
async gauge"] + TQ["TxQ Metrics
async gauge"] + PL["PerfLog RPC / Job
counters + histograms"] + CO["CountedObjects
async gauge"] + LF["Load Factors
async gauge"] end - subgraph newreg["New OTel MetricsRegistry"] - CR["Cache Hit Rates
(async gauge callbacks)"] - TQ["TxQ Metrics
(async gauge callbacks)"] - PL["PerfLog RPC/Job
(counters + histograms)"] - CO["CountedObjects
(async gauge callbacks)"] - LF["Load Factors
(async gauge callbacks)"] + subgraph existing["Pre-existing beast::insight
(unchanged by Phase 9)"] + IN["Node state, PeerFinder,
overlay traffic, caches"] end end subgraph export["Export Pipelines"] + OS["OTel Metrics SDK
PeriodicMetricReader
10s interval"] BI["beast::insight
OTelCollector (Phase 7)"] - OS["OTel Metrics SDK
PeriodicMetricReader"] end - NS --> BI + NS --> OS CR --> OS TQ --> OS PL --> OS CO --> OS LF --> OS + IN --> BI - BI --> OTLP["OTLP/HTTP :4318
/v1/metrics"] - OS --> OTLP + OS --> OTLP["OTLP/HTTP :4318
/v1/metrics"] + BI --> OTLP - style xrpld fill:#1a2633,color:#ccc,stroke:#4a90d9 - style existing fill:#2a4a6b,color:#fff,stroke:#4a90d9 - style newreg fill:#2a4a6b,color:#fff,stroke:#4a90d9 - style export fill:#1a3320,color:#ccc,stroke:#5cb85c - style NS fill:#4a90d9,color:#fff,stroke:#2a6db5 - style CR fill:#5cb85c,color:#fff,stroke:#3d8b3d - style TQ fill:#5cb85c,color:#fff,stroke:#3d8b3d - style PL fill:#5cb85c,color:#fff,stroke:#3d8b3d - style CO fill:#5cb85c,color:#fff,stroke:#3d8b3d - style LF fill:#5cb85c,color:#fff,stroke:#3d8b3d - style BI fill:#449d44,color:#fff,stroke:#2d6e2d - style OS fill:#449d44,color:#fff,stroke:#2d6e2d - style OTLP fill:#f0ad4e,color:#000,stroke:#c78c2e + style xrpld fill:#1a2633,color:#e8e8e8,stroke:#4a90d9 + style newreg fill:#22405c,color:#ffffff,stroke:#5cb85c + style existing fill:#22405c,color:#ffffff,stroke:#4a90d9 + style export fill:#1a3320,color:#e8e8e8,stroke:#5cb85c + style NS fill:#5cb85c,color:#000000,stroke:#3d8b3d + style CR fill:#5cb85c,color:#000000,stroke:#3d8b3d + style TQ fill:#5cb85c,color:#000000,stroke:#3d8b3d + style PL fill:#5cb85c,color:#000000,stroke:#3d8b3d + style CO fill:#5cb85c,color:#000000,stroke:#3d8b3d + style LF fill:#5cb85c,color:#000000,stroke:#3d8b3d + style IN fill:#4a90d9,color:#000000,stroke:#2a6db5 + style OS fill:#449d44,color:#ffffff,stroke:#2d6e2d + style BI fill:#449d44,color:#ffffff,stroke:#2d6e2d + style OTLP fill:#f0ad4e,color:#000000,stroke:#c78c2e ``` -- **beast::insight extensions** (blue): NodeStore I/O metrics added near existing `Database.cpp` registrations — exported via Phase 7's `OTelCollector`. -- **OTel MetricsRegistry** (green): New centralized class using `ObservableGauge` async callbacks for cache, TxQ, PerfLog, CountedObjects, and load factors — polled at 10s intervals by `PeriodicMetricReader`. +- **OTel MetricsRegistry** (green): the single home for every Phase 9 metric — + `ObservableGauge` async callbacks for NodeStore I/O, cache, TxQ, CountedObjects + and load factors, plus synchronous counters/histograms for PerfLog RPC and job + data. Polled at 10s intervals by `PeriodicMetricReader` + (`MetricsRegistry.cpp:289`, `export_interval_millis = 10000`). +- **NodeStore I/O is _not_ a beast::insight extension.** The original plan + routed it through `Database.cpp` insight registrations; the shipped code + registers a `nodestore_state` observable gauge instead + (`MetricsRegistry.cpp:957-965`) that reads `Database`'s public accessors + (`getFetchTotalCount()`, `getStoreDurationUs()`, …). `Database.cpp` has no + `beast::insight` members at all. +- **beast::insight** (blue) still carries the pre-Phase-9 metric surface via + Phase 7's `OTelCollector`; Phase 9 added nothing to it. ### Third-Party Consumer Context @@ -758,67 +840,132 @@ flowchart TB ### Tasks -| Task | Description | -| ---- | ----------------------------------------- | -| 9.1 | NodeStore I/O metrics | -| 9.2 | Cache hit rate metrics + MetricsRegistry | -| 9.3 | TxQ metrics | -| 9.4 | PerfLog per-RPC metrics | -| 9.5 | PerfLog per-job metrics | -| 9.6 | Counted object instance metrics | -| 9.7 | Fee escalation & load factor metrics | -| 9.7a | push_metrics.py parity gauges | -| 9.8 | New Grafana dashboards (2 new, 2 updated) | -| 9.9 | Update documentation | -| 9.10 | Integration tests | +| Task | Description | Status | +| ---- | ------------------------------------------------------- | ----------------------------- | +| 9.1 | NodeStore I/O metrics (`nodestore_state` gauge) | ✅ Done | +| 9.2 | Cache hit rate metrics + `MetricsRegistry` | ✅ Done | +| 9.3 | TxQ metrics | ✅ Done | +| 9.4 | PerfLog per-RPC metrics | ✅ Done | +| 9.5 | PerfLog per-job metrics (`job_type` + `handler` labels) | ✅ Done | +| 9.6 | Counted object instance metrics | ✅ Done | +| 9.7 | Fee escalation & load factor metrics | ✅ Done | +| 9.7a | push_metrics.py parity gauges | ✅ Done | +| 9.8 | New Grafana dashboards (4 new, 2 updated) | ✅ Done | +| 9.9 | Update documentation | ✅ Done | +| 9.9a | Provisioned Grafana alerting (13 rules / 5 groups) | ✅ Done | +| 9.10 | Integration tests / `MetricsRegistry` unit tests | ✅ Done (unit tests) | +| 9.11 | Validator Health dashboard | ✅ Done | +| 9.12 | Peer Quality dashboard | ✅ Done | +| 9.13 | Ledger Economy row on `node-health` | ✅ Done | +| 9.14 | Overlay traffic accounting defects (documentation only) | 📄 Documented, not fixed | +| 9.15 | Peer keepalive / discovery instrumentation | ❌ Not implemented | +| 9.16 | PeerFinder slot and cache metrics | ❌ Not implemented | +| 9.17 | Peer span coverage (`peer.connect` / `peer.message.*`) | ❌ Not implemented (deferred) | -See [Phase9_taskList.md](./Phase9_taskList.md) for detailed per-task breakdown. +See [Phase9_taskList.md](./Phase9_taskList.md) for detailed per-task breakdown, +including the four open items (9.14-9.17) and why each is blocked. + +### Provisioned Grafana Alerting (Task 9.9a) + +Phase 9 also ships the first provisioned Grafana alerting for the OTel stack — +**13 rules in 5 groups**, 2 contact points, and a two-level notification policy +tree, auto-loaded from the existing `provisioning/` mount (no docker-compose +change): + +| File | Contents | +| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| `docker/telemetry/grafana/provisioning/alerting/rules.yaml` | 13 rules across `xrpld-consensus` (3), `xrpld-validator` (2), `xrpld-jobqueue` (3), `xrpld-node-state` (2), `xrpld-overlay` (3) | +| `docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml` | `xrpld-default` (Slack) and `xrpld-critical` (Slack + email) | +| `docker/telemetry/grafana/provisioning/alerting/policies.yaml` | Root route → `xrpld-default`; child route `severity = critical` → `xrpld-critical`. Grouped by `alertname` + `service_instance_id`. | + +Shipped rules: `LedgerHistoryMismatch`, `LedgerCloseStalled`, +`ValidatedLedgerStale`, `ValidationsMissed`, `ValidationsNotChecked`, +`JobQueueTxOverflow`, `JobQueueLatencyHigh`, `NodeStoreIOLatencyHigh`, +`NodeStateFlapping`, `NodeNotFull`, `ManifestJobQueueConvoy`, +`ManifestFloodInbound`, `PeerResourceDisconnects`. Three carry +`severity: critical`, ten `severity: warning`. + +Operator documentation for each alert lives in the **Alerting** section of +`docs/telemetry-runbook.md`. The remaining, genuinely-unshipped rules from the +external-dashboard set are scoped in the appendix under **Task 11.9: Remaining +Alert Rules from External Dashboard**. ### Exit Criteria -- [ ] All ~68 new metrics visible in Prometheus via OTLP pipeline -- [ ] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK -- [ ] 2 new Grafana dashboards operational (Fee Market, Job Queue) -- [ ] No performance regression (< 0.5% CPU overhead from new callbacks) -- [ ] Documentation updated with full new metric inventory +- [ ] All ~68 new metrics visible in Prometheus via OTLP pipeline — every + instrument is registered (`MetricsRegistry.cpp`), but end-to-end + visibility is asserted by the Phase 10 harness, not on this branch +- [x] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK — + covered by `src/tests/libxrpl/telemetry/MetricsRegistry.cpp` + (`async_gauges_start_after_start_is_safe`, + `async_gauges_before_start_does_not_break_start`, + `async_gauges_respect_the_compile_time_guard`, `destructor_calls_stop`, + `disabled_construction`, `disabled_start_stop`, `disabled_recording_methods`) +- [x] 4 new Grafana dashboards operational (Fee Market, Job Queue, Validator + Health, Peer Quality) + 2 updated (Node Health, RPC Performance) — all + present under `docker/telemetry/grafana/dashboards/` +- [ ] No performance regression (< 0.5% CPU overhead from new callbacks) — needs + the Phase 10 benchmark suite; not measured +- [x] Documentation updated with full new metric inventory — + `09-data-collection-reference.md` §5b "Internal Metric Gap Fill (Phase 9)" + and "Phase 9: OTel SDK-Exported Metrics (MetricsRegistry)"; + `docs/telemetry-runbook.md` § Alerting +- [x] Provisioned Grafana alerting shipped (13 rules / 5 groups, 2 contact + points, nested notification policy) --- ## 6.8.3 Phase 10: Synthetic Workload Generation & Telemetry Validation (Weeks 16-17) -> **Status**: In progress. +> **Status**: Implemented on this branch — `docker/telemetry/workload/` (24 +> files) and `.github/workflows/telemetry-validation.yml` are present here. +> Upstream branches do not carry them, so the exit criteria below only hold from +> `pratik/otel-phase10-workload-validation` onward. ### Motivation Before the telemetry stack (Phases 1-9) can be considered production-ready, we need automated proof that all spans, attributes, metrics, Grafana dashboards, and log-trace correlation work correctly under realistic load. This phase establishes a reusable CI-integrated validation suite and performance benchmark baseline. +> **Inventory note**: the "16 spans / 22 attributes / 10 dashboards" figures this +> section used to quote are stale. Do not re-quote fixed counts here — the +> harness hard-codes none of them. `validate_telemetry.py` iterates +> `expected_spans.json` and `expected_metrics.json`, so those two files are the +> only authority, and `grafana_dashboards.uids` in `expected_metrics.json` is the +> authority for dashboards. As of this branch all **15** dashboards on disk +> (`ls docker/telemetry/grafana/dashboards/*.json`) are listed in `uids`, +> `log-derived-insights` included. See +> [Phase10_taskList.md](./Phase10_taskList.md) for the live figures. + ### Architecture -The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Tempo, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage. +The validation uses a **5-node** validator cluster running as native `xrpld` processes (started by `run-full-validation.sh`, `NUM_NODES=5`) alongside a Docker Compose telemetry stack. Only the observability backend runs in containers: `docker-compose.workload.yaml` defines the collector, Tempo, Prometheus, Loki and Grafana, and no `xrpld` service. Five nodes give a real consensus quorum and peer-to-peer span traffic. ```mermaid flowchart LR - subgraph harness["2-Node Validator Cluster (local processes)"] + subgraph harness["5-Node Validator Cluster (native xrpld processes)"] direction TB - V1["Validator 1"] ~~~ V2["Validator 2"] + V1["Validator 1"] ~~~ V2["Validator 2"] ~~~ V3["Validator 3"] + V4["Validator 4"] ~~~ V5["Validator 5"] end subgraph telemetry["Docker Compose Telemetry Stack"] direction TB - COL["OTel Collector
(OTLP + StatsD)"] - JAE["Tempo
(trace search)"] + COL["OTel Collector
(OTLP + filelog)"] + TEMPO["Tempo
(trace search)"] PROM["Prometheus
(metrics)"] + LOKI["Loki
(logs)"] GRAF["Grafana
(dashboards)"] end subgraph generators["Workload Generators"] RPC["RPC Load Generator
(configurable RPS,
command distribution)"] - TX["Transaction Submitter
(10 tx types via
WebSocket command API)"] + TX["Transaction Submitter
(Payment, Offer, NFT,
Escrow, AMM mix)"] end subgraph validation["Validation Suite"] SV["Span Validator
(Tempo API)"] - MV["Metric Validator
(Prometheus API,
all 26 metrics required)"] + MV["Metric Validator
(Prometheus API,
expected_metrics.json)"] + LV["Log-Trace Validator
(Loki API)"] DV["Dashboard Validator
(Grafana API)"] BM["Benchmark Suite
(CPU, memory, latency
ON vs OFF comparison)"] end @@ -833,14 +980,19 @@ flowchart LR style validation fill:#332a1a,color:#ccc,stroke:#f0ad4e style V1 fill:#4a90d9,color:#fff,stroke:#2a6db5 style V2 fill:#4a90d9,color:#fff,stroke:#2a6db5 + style V3 fill:#4a90d9,color:#fff,stroke:#2a6db5 + style V4 fill:#4a90d9,color:#fff,stroke:#2a6db5 + style V5 fill:#4a90d9,color:#fff,stroke:#2a6db5 style COL fill:#4a90d9,color:#fff,stroke:#2a6db5 - style JAE fill:#4a90d9,color:#fff,stroke:#2a6db5 + style TEMPO fill:#4a90d9,color:#fff,stroke:#2a6db5 style PROM fill:#4a90d9,color:#fff,stroke:#2a6db5 + style LOKI fill:#4a90d9,color:#fff,stroke:#2a6db5 style GRAF fill:#4a90d9,color:#fff,stroke:#2a6db5 style RPC fill:#5cb85c,color:#fff,stroke:#3d8b3d style TX fill:#5cb85c,color:#fff,stroke:#3d8b3d style SV fill:#f0ad4e,color:#000,stroke:#c78c2e style MV fill:#f0ad4e,color:#000,stroke:#c78c2e + style LV fill:#f0ad4e,color:#000,stroke:#c78c2e style DV fill:#f0ad4e,color:#000,stroke:#c78c2e style BM fill:#f0ad4e,color:#000,stroke:#c78c2e ``` @@ -850,9 +1002,9 @@ flowchart LR - **Transaction submitter and RPC load generator** both use xrpld's native WebSocket command format (`{"command": ...}`) — not JSON-RPC format. Response data lives inside `"result"` with `"status"` at the top level. - **Node config** requires `[signing_support] true` for server-side signing, and `[ips]` (not `[ips_fixed]`) to ensure peer connections count in `peer_finder_active_*` metrics. - **Metric validation** uses the Prometheus `/api/v1/series` endpoint (not instant queries) to avoid false negatives from stale StatsD gauges. Every metric in `expected_metrics.json` must have > 0 series. -- **StatsD gauge fix**: `StatsDGaugeImpl` initializes `m_dirty = true` so all gauges emit their initial value on first flush. Without this, gauges starting at 0 that never change (e.g. `jobq_job_count`) would be invisible in Prometheus. +- **Gauge visibility**: the harness sets `[insight] server=otel` (`run-full-validation.sh`), so `beast::insight` gauges become OTel observable gauges whose callback is invoked on every collection cycle. A gauge that sits at 0 and never changes (e.g. `jobq_job_count`) therefore still reports, and `/api/v1/series` sees it. - **I/O latency fix**: `io_latency_sampler` emits unconditionally on first sample, then applies the 10 ms threshold. This ensures `ios_latency` is registered in Prometheus even in low-load CI environments. -- **tx.receive span**: Sets default attributes (`xrpl.tx.suppressed = false`, `xrpl.tx.status = "new"`) on span creation so they are always present. The suppressed/bad code paths override these when applicable. +- **tx.receive span**: attribute keys are bare, not dotted — `suppressed` and `tx_status` (`TxSpanNames.h:71,75`). `suppressed` is set on both outcomes (`false` on the accepted path, `true` when the HashRouter suppresses), but `tx_status` is set **only** on the reject/known-bad/dropped paths, so it is absent on a successful receive. Assert on the attribute, not on span status. ### Tasks @@ -868,46 +1020,139 @@ flowchart LR See [Phase10_taskList.md](./Phase10_taskList.md) for detailed per-task breakdown. -### Validation Check Inventory (71 Checks) +### Validation Check Inventory -The validation suite (`validate_telemetry.py`) runs exactly 71 checks, broken down as: +`validate_telemetry.py` derives its check count at run time from +`expected_spans.json` (span types and their required attributes) and +`expected_metrics.json` (metric names and dashboard uids). Per the inventory note +above, no counts are quoted here — those two manifests are the only authority and +any edit to them changes the total. The historical "71 checks" figure predates the +later metric families. The categories are: -- **1 service registration** — `xrpld` exists in Tempo -- **17 span existence** — `rpc.request`, `rpc.process`, `rpc.ws_message`, `rpc.command.*`, `tx.process`, `tx.receive`, `tx.apply`, `consensus.proposal.send`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply`, `ledger.build`, `ledger.validate`, `ledger.store`, `peer.proposal.receive`, `peer.validation.receive` -- **14 span attribute** — required attributes on the 14 spans that define them (22 unique attributes total) -- **2 span hierarchies** — `rpc.process` -> `rpc.command.*`, `ledger.build` -> `tx.apply` (1 skipped: `rpc.request` -> `rpc.process`, cross-thread) -- **1 span duration bounds** — all spans > 0 and < 60 s -- **26 metric existence** — 4 SpanMetrics (`span_calls_total`, `span_duration_milliseconds_{bucket,count,sum}`), 6 StatsD gauges (`ledgermaster_validated_ledger_age`, `published_ledger_age`, `state_accounting_full_duration`, `peer_finder_active_{inbound,outbound}_peers`, `jobq_job_count`), 2 StatsD counters (`rpc_requests_total`, `ledger_fetches_total`), 3 StatsD histograms (`rpc_time`, `rpc_size`, `ios_latency`), 4 overlay traffic (`total_bytes_{in,out}`, `total_messages_{in,out}`), 7 Phase 9 OTLP (`nodestore_state`, `cache_metrics`, `txq_metrics`, `rpc_method_{started,finished}_total`, `object_count`, `load_factor_metrics`) -- **10 dashboard loads** — `rpc-performance`, `transaction-overview`, `consensus-health`, `ledger-operations`, `peer-network`, `node-health`, `network-traffic`, `rpc-pathfinding`, `overlay-traffic-detail`, `ledger-data-sync` +- **Service registration** — `xrpld` exists in Tempo +- **Span existence** — every required entry in `expected_spans.json`. Note that + `rpc.process` is emitted only on the HTTP path (`ServerHandler.cpp`), so under + the harness's WebSocket load it does not appear; it is marked optional. +- **Span attributes** — each span's `required_attributes` +- **Span hierarchies** — the parent/child edges in + `parent_child_relationships`, minus the ones marked `skip` +- **Span duration bounds** — all spans > 0 and < 60 s +- **Metric existence** — every entry in `expected_metrics.json`, queried through + the Prometheus `/api/v1/series` endpoint +- **Dashboard loads** — every uid in `expected_metrics.json` under + `grafana_dashboards.uids` (currently all 15 provisioned dashboards). Note this + only asks Grafana for the dashboard and its panel count; it does not run the + panel queries. +- **Log-trace correlation** — `trace_id` present in Loki plus a Tempo reverse + lookup (skipped in CI via `--skip-loki`, not absent from the suite) -See [Phase10_taskList.md](./Phase10_taskList.md) for the full numbered check-by-check enumeration. +See [Phase10_taskList.md](./Phase10_taskList.md) for the per-task breakdown. -### Current Status +### Known Gaps in CI -**Working** (71/71 checks pass in CI): -All 17 spans, 26 metrics, 10 dashboards, 14 attribute checks, 2 hierarchies, and duration bounds validated. - -**Not implemented or not available in CI**: - -1. `rpc.request` -> `rpc.process` parent-child hierarchy — skipped (cross-thread context propagation) -2. Log-trace correlation validation (Loki) — not included in checks -3. Full 255+ StatsD metric coverage — only 26 representative metrics validated -4. Sustained load / backpressure testing — not implemented -5. `docs/telemetry-runbook.md` updates — not done -6. `09-data-collection-reference.md` "Validation" section — not done -7. **Automated cross-CI baseline persistence** — the regression gate reads a +1. `rpc.process` -> `rpc.command.*` hierarchy — not assertable under the + harness's WebSocket-only load, because `rpc.process` is created only on the + HTTP path. This is a load-shape limitation, not a context-propagation bug. +2. Log-trace correlation — implemented and passing locally; CI passes + `--skip-loki`. +3. Legacy `beast::insight` coverage — `expected_metrics.json` asserts a + representative subset, not all ~270 families. +4. Sustained load / backpressure — the `stress` profile exists in + `workload-profiles.json` but is not wired into CI. +5. **Automated cross-CI baseline persistence** — the regression gate reads a committed baseline; baseline updates flow through a manual PR refresh, not an artifact promoted from `develop` (FU-2). +### CI Deliverable (Task 10.6) + +The Phase 10 CI entry point is `.github/workflows/telemetry-validation.yml` +(367 lines, on the Phase 10 branch). It runs three jobs — `linux-image-tag`, +`build-xrpld`, `validate-telemetry` — and is triggered by `workflow_dispatch` +plus `push` on `pratik/otel-phase*`, `feature/otel-*` and +`feature/telemetry-*`. **There is no cron schedule**, so nothing runs this +workflow on a timer. + +> **Fixed — the `push` trigger's `paths` filter now covers the C++ telemetry +> sources.** The branch filter is only half the trigger; `push` also carries a +> `paths` filter, and it previously read: +> +> ```yaml +> paths: +> - ".github/workflows/telemetry-validation.yml" +> - "docker/telemetry/**" +> - "include/xrpl/basics/Telemetry*.h" # 0 tracked paths +> - "src/xrpld/app/misc/Telemetry*" # 0 tracked paths +> ``` +> +> The last two globs matched **nothing** — neither +> `include/xrpl/basics/Telemetry*.h` nor `src/xrpld/app/misc/Telemetry*` exists. +> The telemetry code lives in `src/xrpld/telemetry/**` (9 files, including +> `MetricsRegistry.cpp`), `src/libxrpl/telemetry/**` (7 files) and +> `include/xrpl/telemetry/**` (10 files), none of which were listed. +> Consequence at the time: a pure C++ telemetry change — new instrument, +> renamed metric, changed span attribute — never triggered this workflow on +> push; only edits under `docker/telemetry/**` or to the workflow file itself +> did. +> +> The two dead globs have been replaced with the three real module directories, +> so the filter now reads: +> +> ```yaml +> paths: +> - ".github/workflows/telemetry-validation.yml" +> - "docker/telemetry/**" +> - "include/xrpl/telemetry/**" +> - "src/libxrpl/telemetry/**" +> - "src/libxrpl/beast/insight/**" +> - "src/xrpld/telemetry/**" +> ``` +> +> `src/libxrpl/beast/insight/**` is included because it holds `OTelCollector.cpp`, +> the `beast::insight` OTLP export path the harness depends on. Residual gap: the +> instrumented call sites scattered through `src/xrpld/app/` are not listed, so a +> change that only adds or moves a span at a call site does not trigger the +> workflow on push. Those are reachable by manual dispatch. + +> **Caveat — four inert inputs (documented, not wired).** The workflow declares +> five `workflow_dispatch` inputs, but only `run_benchmark` changes behaviour. +> `rpc_rate`, `rpc_duration`, `tx_tps` and `tx_duration` are forwarded as +> `--rpc-rate` / `--rpc-duration` / `--tx-tps` / `--tx-duration` to +> `run-full-validation.sh`, which parses them into shell variables and then +> never reads them again: load shape comes entirely from +> `--profile` / `workload-profiles.json` (the orchestrator is invoked with +> `--profile` only). Changing those four inputs has no effect on the generated +> workload. +> +> Resolution taken: each of the four now carries +> `description: "UNUSED — has no effect. Load shape comes from the workload +profile."`, and a comment above the `inputs:` block plus one at the +> ARGS-building step record why they are kept. They were **labelled, not +> wired**, because wiring them would be a behaviour change: the orchestrator is +> profile-driven, so honouring them means either synthesising a temporary +> profile or reintroducing the pre-profile single-phase load path. That belongs +> in its own change, not in a docs-accuracy pass. The alternative — deleting the +> inputs — would break saved dispatch input sets for no gain. + ### Exit Criteria -- [x] 2-node validator cluster starts and reaches consensus -- [x] Validation suite confirms all required spans, attributes, and metrics (71/71 checks) -- [x] All 10 Grafana dashboards render data -- [ ] Benchmark shows < 3% CPU overhead, < 5MB memory overhead +- [x] 5-node validator cluster starts and reaches consensus — note that + `docker-compose.workload.yaml` contains only the observability backend + (collector, Tempo, Prometheus, Loki, Grafana); the 5 validators are native + `xrpld` processes started by `run-full-validation.sh` (`NUM_NODES=5`) +- [x] Validation suite confirms the full span / attribute / metric inventory + (counts computed dynamically from `expected_spans.json` and + `expected_metrics.json`) +- [x] All 15 provisioned Grafana dashboards are asserted to load — every uid on + disk is now listed in `grafana_dashboards.uids`. Caveat: the check is + load-and-panel-count only, so it does not prove every panel returns data +- [ ] Benchmark shows < 3% CPU overhead, < 5MB memory overhead — needs a + measured run - [x] CI workflow runs validation on telemetry branch changes -- [x] OTel-driven regression gate: captures per-span/per-RPC/per-job timings - from Prometheus and compares against a committed baseline + (`.github/workflows/telemetry-validation.yml`) +- [x] OTel-driven regression gate: captures per-span and per-job timings from + Prometheus and compares against a committed baseline. Per-RPC timings are + **not** gated — `regression-metrics.json` defines only `spans` and + `job_queue` groups (FU-4). --- @@ -1162,7 +1407,8 @@ flowchart TB - ~~Validator list and manifest tracing~~ — descoped - ~~Amendment voting tracing~~ — descoped - ~~SHAMap sync tracing~~ — descoped -- Full end-to-end traces (client → RPC → TX → consensus → ledger) — partial (tx-consensus correlation not yet done) +- Full end-to-end traces (client → RPC → TX → consensus → ledger) — tx-consensus + correlation shipped as `tx.included` events in `doAccept` (Task 4.6) **Code Changes**: ~100 lines across 3 consensus files @@ -1203,13 +1449,13 @@ Clear, measurable criteria for each phase. ### 6.12.1 Phase 1: Core Infrastructure -| Criterion | Measurement | Target | -| --------------- | ---------------------------------------------------------- | ---------------------------- | -| SDK Integration | `cmake --build` succeeds with `-DXRPL_ENABLE_TELEMETRY=ON` | ✅ Compiles | -| Runtime Toggle | `enabled=0` produces zero overhead | <0.1% CPU difference | -| Span Creation | Unit test creates and exports span | Span appears in Tempo | -| Configuration | All config options parsed correctly | Config validation tests pass | -| Documentation | Developer guide exists | PR approved | +| Criterion | Measurement | Target | +| --------------- | ---------------------------------------------- | ---------------------------- | +| SDK Integration | `cmake --build` succeeds with `-Dtelemetry=ON` | ✅ Compiles | +| Runtime Toggle | `enabled=0` produces zero overhead | <0.1% CPU difference | +| Span Creation | Unit test creates and exports span | Span appears in Tempo | +| Configuration | All config options parsed correctly | Config validation tests pass | +| Documentation | Developer guide exists | PR approved | **Definition of Done**: All criteria met, PR merged, no regressions in CI. @@ -1267,19 +1513,19 @@ Clear, measurable criteria for each phase. ### 6.12.6 Success Metrics Summary -| Phase | Primary Metric | Secondary Metric | Deadline | Status | -| -------- | ------------------------------------------------------------------ | --------------------------- | -------------- | ------------------ | -| Phase 1 | SDK compiles and runs | Zero overhead when disabled | End of Week 2 | Active | -| Phase 2 | 100% RPC coverage | <1ms latency overhead | End of Week 4 | Active | -| Phase 3 | Cross-node traces work | <5% throughput impact | End of Week 6 | Active | -| Phase 4 | Consensus fully traced | No consensus timing impact | End of Week 8 | Active | -| Phase 5 | Production deployment | Operators trained | End of Week 9 | Active | -| Phase 6 | StatsD metrics in Prometheus | 3 dashboards operational | End of Week 10 | Active | -| Phase 7 | All metrics via OTLP | No StatsD dependency | End of Week 12 | Active | -| Phase 8 | trace_id in logs + Loki | Tempo↔Loki correlation | End of Week 13 | Active | -| Phase 9 | 68+ new internal metrics in Prom | 2 new dashboards | End of Week 15 | Future Enhancement | -| Phase 10 | Full telemetry stack validated; OTel-sourced regression gate in CI | < 3% CPU overhead proven | End of Week 17 | Future Enhancement | -| Phase 11 | Third-party metrics via receiver | 4 new dashboards + alerting | End of Week 20 | Future Enhancement | +| Phase | Primary Metric | Secondary Metric | Deadline | Status | +| -------- | ------------------------------------------------------------------ | --------------------------------------------- | -------------- | ------------------ | +| Phase 1 | SDK compiles and runs | Zero overhead when disabled | End of Week 2 | Active | +| Phase 2 | 100% RPC coverage | <1ms latency overhead | End of Week 4 | Active | +| Phase 3 | Cross-node traces work | <5% throughput impact | End of Week 6 | Active | +| Phase 4 | Consensus fully traced | No consensus timing impact | End of Week 8 | Active | +| Phase 5 | Production deployment | Operators trained | End of Week 9 | Active | +| Phase 6 | StatsD metrics in Prometheus | 3 dashboards operational | End of Week 10 | Active | +| Phase 7 | All metrics via OTLP | No StatsD dependency | End of Week 12 | Active | +| Phase 8 | trace_id in logs + Loki | Tempo↔Loki correlation | End of Week 13 | Active | +| Phase 9 | 68+ new internal metrics in Prom | 4 new dashboards + 13 provisioned alert rules | End of Week 15 | Complete | +| Phase 10 | Full telemetry stack validated; OTel-sourced regression gate in CI | < 3% CPU overhead proven | End of Week 17 | On Phase 10 branch | +| Phase 11 | Third-party metrics via receiver | 4 new dashboards + 14 remaining alert rules | End of Week 20 | Not started | --- @@ -1374,7 +1620,6 @@ flowchart TB > **Date**: 2026-03-30 > **Status**: Draft > **Source**: [realgrapedrop/xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard) -> **Jira Epic**: RIPD-5060 ### Summary @@ -1403,7 +1648,7 @@ Integrate 29 missing metrics, 18 alert rules, and enriched span attributes from | Upgrade Awareness | `peers_higher_version_pct`, `upgrade_recommended` | 2 | | Storage / Other | `ledger_nudb_bytes`, `jq_trans_overflow_total`, `initial_sync_duration_seconds` | 3 | -#### Alert Rules (18 total, from external dashboard) +#### Alert Rules (18 in the external dashboard; 4 addressed by Phase 9 — 2 fully, 2 partially) | Group | Count | Rules | | ----------- | ----- | ----------------------------------------------------------------------------------------------------------------------- | @@ -1411,6 +1656,12 @@ Integrate 29 missing metrics, 18 alert rules, and enriched span attributes from | Network | 3 | Peer drop >10%/30%, P90 latency + disconnect correlation | | Performance | 7 | CPU >80%, memory >90%, disk >85%, job queue overflow, upgrade recommended, tx rate drop, stale ledger | +> Phase 9 ships **13 provisioned rules in 5 groups** against xrpld's own metric +> surface; 4 of them address external rules — **fully** for unhealthy state and +> job queue overflow, only **partially** for IO latency and stale ledger (looser +> thresholds and longer windows; see the coverage table under Task 11.9). The 14 +> genuinely-remaining rules are scoped under Task 11.9 below. + --- ### Branch-to-Change Mapping @@ -1423,20 +1674,37 @@ Integrate 29 missing metrics, 18 alert rules, and enriched span attributes from Add node-level health context to every `rpc.command.*` span so operators can correlate RPC behavior with node state. -New span attributes on `rpc.command.*`: +> **Status: NOT IMPLEMENTED as span attributes.** Neither key was ever added to +> a span. The dotted `xrpl.*` **span-attribute** namespace was dropped in favour +> of bare/underscore keys (`9e27120a15`), and these two were never re-added under +> any name. Falsifiable check: `grep -rn 'seg::xrpl' src/ include/` → exactly **2** +> hits, both in `include/xrpl/telemetry/SpanNames.h:117-118` +> (`attr::networkId` / `attr::networkType`, i.e. `xrpl.network.id` and +> `xrpl.network.type`), and both are **resource** attributes set on the OTel +> resource at startup, not span attributes. (Do not use +> `grep 'makeStr("xrpl\.'` as evidence — the keys were always composed with +> `join(seg::…)`, never that literal, so it has returned 0 hits since day one and +> proves nothing.) +> The **values** are exported instead as `MetricsRegistry` metric label values: +> `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1014`) and +> `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1216`). +> Correlating an RPC with node state therefore requires a metric join, not a +> span filter. Kept here as an open item. -| Attribute | Type | Source | Value Example | -| ----------------------------- | ------ | ------------------------------------ | --------------------- | -| `xrpl.node.amendment_blocked` | bool | `app_.getOPs().isAmendmentBlocked()` | `true` | -| `xrpl.node.server_state` | string | `app_.getOPs().strOperatingMode()` | `"full"`, `"syncing"` | +Proposed (never built) span attributes on `rpc.command.*`: + +| Attribute (proposed) | Type | Source | Value Example | Status | +| -------------------- | ------ | ------------------------------------ | --------------------- | ---------------------------------------------- | +| `amendment_blocked` | bool | `app_.getOPs().isAmendmentBlocked()` | `true` | ❌ Never implemented — metric label value only | +| `server_state` | string | `app_.getOPs().strOperatingMode()` | `"full"`, `"syncing"` | ❌ Never implemented — metric label value only | **File**: `src/xrpld/rpc/detail/RPCHandler.cpp` (in the `rpc.command.*` span creation block, after existing setAttribute calls) -**Rationale**: RPC is the operator's primary interaction point. When a node is amendment-blocked or degraded, every RPC response is suspect. Tagging spans with this state enables Jaeger queries like `{name=~"rpc.command.*"} | xrpl.node.amendment_blocked = true` to find all RPCs served during a blocked period. +**Rationale**: RPC is the operator's primary interaction point. When a node is amendment-blocked or degraded, every RPC response is suspect. Tagging spans with this state would enable TraceQL queries like `{name=~"rpc.command.*" && span.amendment_blocked = true}` to find all RPCs served during a blocked period. **Exit Criteria**: -- [ ] `rpc.command.server_info` spans carry `xrpl.node.amendment_blocked` and `xrpl.node.server_state` attributes +- [ ] `rpc.command.server_info` spans carry `amendment_blocked` and `server_state` attributes — **open**, never implemented - [ ] No measurable latency impact (attribute values are cached atomics, not computed per-call) --- @@ -1451,18 +1719,27 @@ Add the relaying peer's xrpld version to transaction receive spans to enable ver New span attribute on `tx.receive`: -| Attribute | Type | Source | Value Example | -| ------------------- | ------ | -------------------- | --------------- | -| `xrpl.peer.version` | string | `peer->getVersion()` | `"xrpld-2.4.0"` | +| Attribute | Type | Source | Value Example | Defined at | +| -------------- | ------ | -------------------- | --------------- | ------------------ | +| `peer_version` | string | `peer->getVersion()` | `"xrpld-2.4.0"` | `TxSpanNames.h:79` | -**File**: `src/xrpld/overlay/detail/PeerImp.cpp` (in the `tx.receive` span block, after existing `xrpl.peer.id` setAttribute) +> The dotted `xrpl.peer.version` form in the original spec was never emitted; the +> live key is the bare `peer_version` (`9e27120a15` dropped the `xrpl.*` +> namespace repo-wide). + +**File**: `src/xrpld/overlay/detail/PeerImp.cpp` (in the `tx.receive` span block, after the existing `peer_id` setAttribute) **Rationale**: Transaction relay is where version mismatches cause subtle serialization or validation bugs. Tracing "this tx came from a v2.3.0 peer" helps diagnose compatibility issues during network upgrades. **Exit Criteria**: -- [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string -- [ ] Attribute is omitted (not empty-string) when `getVersion()` returns empty +- [x] `tx.receive` spans carry `peer_version` attribute with a non-empty version + string — `PeerImp.cpp:1341-1342` sets `tx_span::attr::peerVersion` on the + `txReceiveSpan` created at `:1330` +- [x] Attribute is omitted (not empty-string) when `getVersion()` returns empty — + the call site is guarded: + `if (auto const version = getVersion(); !version.empty())` + (`PeerImp.cpp:1341`), so no attribute is set at all on the empty path --- @@ -1474,26 +1751,37 @@ New span attribute on `tx.receive`: Add ledger hash and validation type to validation spans on both send and receive paths. This enables trace-level agreement analysis — filter by ledger hash to see which validators agreed. -New span attributes on `consensus.validation.send`: +> **Status: SHIPPED**, with one exception noted below. All keys are bare / +> underscore — the dotted `xrpl.*` forms in the original spec were never emitted +> as **span** attributes. Check: `grep -rn 'seg::xrpl' src/ include/` → 2 hits, +> both `SpanNames.h:117-118` resource attributes (`xrpl.network.{id,type}`). -| Attribute | Type | Source | Value Example | -| ----------------------------- | ------ | --------------------------------------- | --------------------------- | -| `xrpl.validation.ledger_hash` | string | Ledger hash from `validate()` call args | `"A1B2C3..."` (64-char hex) | -| `xrpl.validation.full` | bool | Whether this is a full validation | `true` | +Span attributes on `consensus.validation.send` (`RCLConsensus.cpp:975-981`): -New span attributes on `peer.validation.receive`: +| Attribute | Type | Source | Value Example | Defined at | +| ----------------- | ------ | --------------------------------------- | --------------------------- | ----------------- | +| `ledger_hash` | string | Ledger hash from `validate()` call args | `"A1B2C3..."` (64-char hex) | `SpanNames.h:147` | +| `full_validation` | bool | Whether this is a full validation | `true` | `SpanNames.h:148` | +| `ledger_seq` | int64 | `ledger.seq()` | `93110248` | shared consensus | +| `proposing` | bool | `proposing` argument | `true` | shared consensus | -| Attribute | Type | Source | Value Example | -| ---------------------------------- | ------ | ------------------------------------- | --------------------------- | -| `xrpl.peer.validation.ledger_hash` | string | From deserialized STValidation object | `"A1B2C3..."` (64-char hex) | -| `xrpl.peer.validation.full` | bool | From STValidation flags | `true` | +Span attributes on `peer.validation.receive` (`PeerImp.cpp:2573-2574`): -New span attributes on `consensus.accept`: +| Attribute | Type | Source | Value Example | Defined at | +| ----------------- | ------ | ------------------------------------- | --------------------------- | -------------------- | +| `ledger_hash` | string | From deserialized STValidation object | `"A1B2C3..."` (64-char hex) | `PeerSpanNames.h:35` | +| `full_validation` | bool | `val->isFull()` | `true` | `PeerSpanNames.h:34` | -| Attribute | Type | Source | Value Example | -| ------------------------------------ | ----- | ---------------------------------------- | ------------- | -| `xrpl.consensus.validation_quorum` | int64 | `app_.validators().quorum()` | `28` | -| `xrpl.consensus.proposers_validated` | int64 | `result.proposers` from consensus result | `35` | +Span attributes on `consensus.accept`: + +| Attribute | Type | Source | Value Example | Status | +| --------------------- | ----- | ---------------------------------------- | ------------- | ------------------------------------------------------------------- | +| `quorum` | int64 | `app_.getValidators().quorum()` | `28` | ✅ `RCLConsensus.cpp:516`, `ConsensusSpanNames.h:219` | +| `proposers_validated` | int64 | `result.proposers` from consensus result | `35` | ❌ **Never implemented** — no attribute of this name exists in code | + +> `proposers` is already set on `consensus.accept` (`RCLConsensus.cpp:513`), so a +> separate `proposers_validated` key would be a duplicate under a different +> name; that is why it was never added. It stays open only as a naming decision. **Files**: @@ -1504,10 +1792,11 @@ New span attributes on `consensus.accept`: **Exit Criteria**: -- [ ] `consensus.validation.send` spans carry `xrpl.validation.ledger_hash` and `xrpl.validation.full` -- [ ] `peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full` -- [ ] `consensus.accept` spans carry `xrpl.consensus.validation_quorum` and `xrpl.consensus.proposers_validated` -- [ ] Ledger hash attributes match between send and receive for the same ledger +- [x] `consensus.validation.send` spans carry `ledger_hash` and `full_validation` — `RCLConsensus.cpp:975-981` +- [x] `peer.validation.receive` spans carry `ledger_hash` and `full_validation` — `PeerImp.cpp:2573-2574` +- [x] `consensus.accept` spans carry `quorum` — `RCLConsensus.cpp:516` +- [ ] `consensus.accept` spans carry `proposers_validated` — **open**, never implemented (see note above) +- [ ] Ledger hash attributes match between send and receive for the same ledger — needs a live multi-node run --- @@ -1702,7 +1991,7 @@ New MetricsRegistry observable gauge for node state duration. | Gauge Name | Label `metric=` | Type | Source | | ---------------- | ------------------------------- | ------ | ------------------------------------------------ | -| `state_tracking` | `state_value` | int64 | 0-7 numeric encoding matching external dashboard | +| `state_tracking` | `state_value` | double | 0-6 numeric encoding matching external dashboard | | | `time_in_current_state_seconds` | double | `now - lastModeChangeTime` | **State value encoding**: @@ -1813,9 +2102,12 @@ Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats. > **Ref**: Adds to existing Phase 9 task list. Depends on Phase 7 gauges/counters. Consumed by Phase 10 (dashboard load checks). -**Task 9.11: Validator Health Dashboard** +**Task 9.11: Validator Health Dashboard** — ✅ shipped -New Grafana dashboard: `validator-health.json` +New Grafana dashboard: `validator-health.json` (uid `validator-health`). The +shipped dashboard has **17 panels** across 3 rows — Validation Agreement, +Validation Rates, Server State & Consensus — i.e. 4 more than the 13 planned +below. | Panel | Type | PromQL | | -------------------------- | ---------- | -------------------------------------------------------- | @@ -1837,7 +2129,7 @@ New Grafana dashboard: `validator-health.json` --- -**Task 9.12: Peer Quality Dashboard** +**Task 9.12: Peer Quality Dashboard** — ✅ shipped (6 panels, uid `peer-quality`) New Grafana dashboard: `peer-quality.json` @@ -1852,9 +2144,10 @@ New Grafana dashboard: `peer-quality.json` --- -**Task 9.13: Ledger Economy Dashboard Panels** +**Task 9.13: Ledger Economy Dashboard Panels** — ✅ shipped -Add a "Ledger Economy" row to the existing `node-health.json` dashboard: +The "Ledger Economy" row is present on `node-health.json` with all 5 +`ledger_economy` panels: | Panel | Type | PromQL | | -------------------- | ---------- | --------------------------------------------- | @@ -1874,18 +2167,23 @@ Add a "Ledger Economy" row to the existing `node-health.json` dashboard: Add checks to `validate_telemetry.py` for all new span attributes and metrics. -**New span attribute checks (~8)**: +**New span attribute checks** — bare/underscore keys; the dotted `xrpl.*` forms +were never emitted: -| Span Name | New Attribute | -| --------------------------- | ------------------------------------ | -| `rpc.command.server_info` | `xrpl.node.amendment_blocked` | -| `rpc.command.server_info` | `xrpl.node.server_state` | -| `tx.receive` | `xrpl.peer.version` | -| `consensus.validation.send` | `xrpl.validation.ledger_hash` | -| `consensus.validation.send` | `xrpl.validation.full` | -| `peer.validation.receive` | `xrpl.peer.validation.ledger_hash` | -| `consensus.accept` | `xrpl.consensus.validation_quorum` | -| `consensus.accept` | `xrpl.consensus.proposers_validated` | +| Span Name | New Attribute | Emitted? | +| --------------------------- | --------------------- | ---------------------------------------------- | +| `rpc.command.server_info` | `amendment_blocked` | ❌ never implemented — metric label value only | +| `rpc.command.server_info` | `server_state` | ❌ never implemented — metric label value only | +| `tx.receive` | `peer_version` | ✅ `TxSpanNames.h:79` | +| `consensus.validation.send` | `ledger_hash` | ✅ `RCLConsensus.cpp:975-981` | +| `consensus.validation.send` | `full_validation` | ✅ `RCLConsensus.cpp:975-981` | +| `peer.validation.receive` | `ledger_hash` | ✅ `PeerImp.cpp:2573` | +| `peer.validation.receive` | `full_validation` | ✅ `PeerImp.cpp:2574` | +| `consensus.accept` | `quorum` | ✅ `RCLConsensus.cpp:516` | +| `consensus.accept` | `proposers_validated` | ❌ never implemented | + +Only the ✅ rows are checkable; the ❌ rows must not be added to +`expected_spans.json` as required attributes. **New metric existence checks (~13)**: @@ -1920,9 +2218,10 @@ Add checks to `validate_telemetry.py` for all new span attributes and metrics. | `validation_agreement_pct_1h` | in [0, 100] | | `unl_expiry_days` | > 0 (not expired) | | `peer_latency_p90_ms` | > 0 (peers exist) | -| `state_value` | in [0, 7] | +| `state_value` | in [0, 6] | -**Total new checks: ~28** (bringing total from 73 to ~101) +**Total new checks: ~28** — the harness computes its check total dynamically, so +no fixed "N of N" figure is asserted here. --- @@ -1930,50 +2229,89 @@ Add checks to `validate_telemetry.py` for all new span attributes and metrics. > **Ref**: Adds to existing Phase 11 task list. Depends on Phase 7 metrics and Phase 9 dashboards. -**Task 11.9: Alert Rules from External Dashboard** +**Task 11.9: Remaining Alert Rules from External Dashboard** -Port 18 alert rules from the external `xrpl-validator-dashboard` to Grafana alerting provisioning. +> **Ownership correction.** Provisioned Grafana alerting is **not** a Phase 11 +> deliverable and does **not** live at +> `docker/telemetry/grafana/alerting/{alert-rules,contact-points,notification-policies}.yaml` +> — that directory has never existed. It shipped on **Phase 9** (`7cabf91a0d`) +> at `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` +> with **13 rules in 5 groups**, 2 contact points and a nested notification +> policy. See §6.8.2 → "Provisioned Grafana Alerting (Task 9.9a)". -**Critical Group** (8 rules, eval interval 10s): +Of the 18 external-dashboard rules originally listed here, **4 are addressed** by +the Phase 9 set (under different names and against xrpld's own metric surface) — +but only 2 of those 4 are a like-for-like match. The other 2 are **partially +covered**: the Phase 9 rule watches the same failure mode at a materially looser +threshold and a longer `for` window, so the external rule's sensitivity is _not_ +reproduced. -| Rule | Condition | For | -| ------------------- | ------------------------------------------------------- | --- | -| Agreement Below 90% | `validation_agreement{metric="agreement_pct_24h"} < 90` | 30s | -| Not Proposing | `state_tracking{metric="state_value"} < 6` | 10s | -| Unhealthy State | `state_tracking{metric="state_value"} < 4` | 10s | -| Amendment Blocked | `validator_health{metric="amendment_blocked"} == 1` | 1m | -| UNL Expiring | `validator_health{metric="unl_expiry_days"} < 14` | 1h | -| High IO Latency | `histogram_quantile(0.95, ios_latency_bucket) > 50` | 1m | -| High Load Factor | `load_factor_metrics{metric="load_factor"} > 1000` | 1m | -| Peer Count Critical | `server_info{metric="peers"} < 5` | 1m | +| External rule | Addressed by (Phase 9 rule) | Group | Coverage | +| ------------------ | ---------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Unhealthy State | `NodeNotFull` | `xrpld-node-state` | Full | +| High IO Latency | `NodeStoreIOLatencyHigh` (`ios_latency_milliseconds_bucket` p95) | `xrpld-jobqueue` | **Partial** — Phase 9 fires at p95 **> 1000 ms for 10m**; the external rule fires at **> 50 for 1m**. A 20× looser threshold and a 10× longer window | +| Job Queue Overflow | `JobQueueTxOverflow` (`jq_trans_overflow_total`) | `xrpld-jobqueue` | Full | +| Stale Ledger | `ValidatedLedgerStale` (`ledgermaster_validated_ledger_age`) | `xrpld-consensus` | **Partial** — different metric and threshold: Phase 9 uses `ledgermaster_validated_ledger_age > 60` for 5m; the external rule uses `ledger_economy{ledger_age_seconds} > 30` for 1m | -**Network Group** (3 rules, eval interval 10s): +> The two partial rows are **not** closed by Phase 9. Either re-baseline the +> Phase 9 thresholds against the measured evidence, or add the tighter external +> variants alongside them under Task 11.12 — do not treat them as done. -| Rule | Condition | For | -| ------------------------- | ----------------------------------------------------------- | --- | -| Peer Drop >10% | `delta(server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s | -| Peer Drop >30% | Same formula, threshold -30 | 30s | -| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0` | 2m | +Phase 9 additionally ships 9 rules with no external counterpart: +`LedgerHistoryMismatch`, `LedgerCloseStalled`, `ValidationsMissed`, +`ValidationsNotChecked`, `JobQueueLatencyHigh`, `NodeStateFlapping`, +`ManifestJobQueueConvoy`, `ManifestFloodInbound`, `PeerResourceDisconnects`. -**Performance Group** (7 rules, eval interval 10s): +**Remaining open work for Phase 11 — 14 rules that genuinely do not exist yet:** -| Rule | Condition | For | -| ------------------- | ------------------------------------------------------ | --- | -| CPU High | Per-core CPU > 80% | 2m | -| Memory Critical | Memory usage > 90% | 1m | -| Disk Warning | Disk usage > 85% | 2m | -| Job Queue Overflow | `rate(jq_trans_overflow_total[5m]) > 0` | 1m | -| Upgrade Recommended | `peer_quality{metric="peers_higher_version_pct"} > 60` | 1m | -| TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m | -| Stale Ledger | `ledger_economy{metric="ledger_age_seconds"} > 30` | 1m | +**Critical** (6 remaining): -**Notification channels**: Template configs for Email/SMTP, Discord, Slack, PagerDuty. +| Rule | Condition | Blocked on | +| ------------------- | ------------------------------------------------------- | ---------- | +| Agreement Below 90% | `validation_agreement{metric="agreement_pct_24h"} < 90` | — | +| Not Proposing | `state_tracking{metric="state_value"} < 6` | — | +| Amendment Blocked | `validator_health{metric="amendment_blocked"} == 1` | — | +| UNL Expiring | `validator_health{metric="unl_expiry_days"} < 14` | — | +| High Load Factor | `load_factor_metrics{metric="load_factor"} > 1000` | — | +| Peer Count Critical | `server_info{metric="peers"} < 5` | — | -**Files**: +> **"Not Proposing" is unblocked.** The `state_tracking` gauge **is** +> implemented: `MetricsRegistry::registerStateTrackingGauge()` +> (`MetricsRegistry.cpp:1461-1510`) creates +> `CreateDoubleObservableGauge("state_tracking", …)` at `:1466` and observes +> `state_value` (`:1497`) and `time_in_current_state_seconds` (`:1502`). It is +> already consumed by `validator-health.json:765,971` and +> `ledger-data-sync.json:869`, and documented in +> [09-data-collection-reference.md](./09-data-collection-reference.md) § +> "State Tracking". Only **3** of the 14 remaining rules are blocked on anything — +> CPU High, Memory Critical and Disk Warning, all needing `node_exporter`. -- `docker/telemetry/grafana/alerting/alert-rules.yaml` (new or extend existing) -- `docker/telemetry/grafana/alerting/contact-points.yaml` -- `docker/telemetry/grafana/alerting/notification-policies.yaml` +**Network** (3 remaining): + +| Rule | Condition | +| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Peer Drop >10% | `delta(server_info{metric="peers"}[30s]) / ... * 100 < -10` | +| Peer Drop >30% | Same formula, threshold -30 | +| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0` — partially covered by `PeerResourceDisconnects`, which has no latency term | + +**Performance** (5 remaining): + +| Rule | Condition | Blocked on | +| ------------------- | ------------------------------------------------------ | ---------------------------------------- | +| CPU High | Per-core CPU > 80% | needs `node_exporter` — not in the stack | +| Memory Critical | Memory usage > 90% | needs `node_exporter` | +| Disk Warning | Disk usage > 85% | needs `node_exporter` | +| Upgrade Recommended | `peer_quality{metric="peers_higher_version_pct"} > 60` | — | +| TX Rate Drop | Transaction rate dropped > 50% in 5m window | — | + +**Notification channels**: the shipped `contactpoints.yaml` provides Slack and +email. Templates for Discord and PagerDuty remain open. + +**Files** (extend the Phase 9 location; do **not** create a second `alerting/` tree): + +- `docker/telemetry/grafana/provisioning/alerting/rules.yaml` (add groups) +- `docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml` (add receivers) +- `docker/telemetry/grafana/provisioning/alerting/policies.yaml` (add routes) --- @@ -1991,18 +2329,44 @@ Document the external dashboard's "fast path" pattern as a future optimization f ### Documentation Updates -#### `docs/telemetry-runbook.md` (on Phase 9 branch) +#### `docs/telemetry-runbook.md` (on Phase 9 branch) — partially done -Add new sections after "Phase 9: OTel Metrics Alerting Rules": +- [x] **Alerting** section — shipped; documents all 13 provisioned rules, + thresholds, likely causes, and how to point a contact point at a real + receiver. + Six dashboard reference sections remain unwritten (`fee-market`, `job-queue`, + `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality`, + `validator-health`), plus one operator explainer: -1. **Validator Health Monitoring** — explains agreement tracking, amendment blocked, UNL expiry, with example PromQL queries -2. **Peer Quality Monitoring** — explains P90 latency, insane peers, version awareness -3. **Ledger Economy Monitoring** — explains fee/reserve gauges, transaction rate, ledger age -4. **Validation Agreement Explained** — operator-facing explanation of the reconciliation algorithm (8s grace, 5m late repair), what "missed" means, and when to worry +- [ ] **`validator-health` guide** — explains agreement tracking, amendment blocked, UNL expiry, with example PromQL queries +- [ ] **`peer-quality` guide** — explains P90 latency, insane peers, version awareness +- [ ] **`fee-market` guide** — explains TxQ depth vs capacity, fee escalation levels, load factor breakdown +- [ ] **`job-queue` guide** — explains per-job-type rates, queue wait vs execution time, concurrency limits +- [ ] **`ledger-data-sync` guide** — explains sync state, ledger acquisition, I/O latency +- [ ] **`overlay-traffic-detail` guide** — explains per-category traffic accounting (note the §6 defects that flatline some panels) +- [ ] **Validation Agreement Explained** — operator-facing explanation of the reconciliation algorithm (8s grace, 5m late repair), what "missed" means, and when to worry -#### `OpenTelemetryPlan/09-data-collection-reference.md` (on Phase 9 branch) +> Ledger economy is a **row on `node-health`**, not a dashboard of its own, so it +> falls under that already-documented section rather than the six above. -Add new metric tables in a "Phase 7+: External Dashboard Parity" section covering all 29 new metrics with their gauge names, label values, types, and sources. +> Still open. The runbook itself records the gap at its dashboard reference +> section, and it names **six** dashboards, not four: "Nine dashboards have a +> reference section below. `fee-market`, `job-queue`, `ledger-data-sync`, +> `overlay-traffic-detail`, `peer-quality`, and `validator-health` are +> provisioned but not yet documented here — their panel descriptions carry the +> same six-heading reference format, so open the panel info icon in Grafana until +> a section is written." (15 dashboards on disk − 6 undocumented = 9 documented.) +> So the remaining runbook work is **six** dashboard guides, plus the Validation +> Agreement explainer listed above. + +#### `OpenTelemetryPlan/09-data-collection-reference.md` (on Phase 9 branch) — done + +- [x] "Phase 7+: External Dashboard Parity Metrics" section with gauge names, + label values, types and sources. +- [x] §5b "Internal Metric Gap Fill (Phase 9)" and "Phase 9: OTel SDK-Exported + Metrics (MetricsRegistry)". +- [x] "New Grafana Dashboards (Phase 9)" and "Updated Grafana Dashboards + (Phase 9)" reference tables. --- @@ -2018,11 +2382,13 @@ Phase 6 (StatsD bridge: peerDisconnectsCharges) │ Phase 7 (ValidationTracker + 7 gauges + 7 counters + agreement gauge) │ -Phase 9 (3 dashboards + ledger economy panels + runbook + data-collection-ref) +Phase 9 (4 new dashboards + ledger economy panels + 13 provisioned + alert rules + data-collection-ref; runbook Alerting only) │ -Phase 10 (28 new validation checks in validate_telemetry.py) +Phase 10 (new validation checks in validate_telemetry.py + + .github/workflows/telemetry-validation.yml) │ -Phase 11 (18 alert rules + dual-datasource docs) +Phase 11 (14 remaining alert rules + dual-datasource docs) ``` ### Rebase Strategy diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index 4ebb6028fd..daa7fa9693 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -17,14 +17,33 @@ ### Quick Start with Tempo ```bash -# Start Tempo with OTLP support +# Start Tempo with OTLP support. +# Version pinned to match docker/telemetry/docker-compose.yml:55 — keep the +# two in step, since Tempo config keys change between minor releases. +# +# Only 4317 (OTLP/gRPC) is published: docker/telemetry/tempo.yaml:28-33 +# declares a single distributor receiver, `otlp.protocols.grpc` on +# 0.0.0.0:4317. There is no `http` protocol block, so nothing listens on 4318 +# and publishing it would give you a port that silently refuses connections. +# 3200 is Tempo's HTTP API/query port (tempo.yaml:17-18), not an ingest port. docker run -d --name tempo \ -p 3200:3200 \ -p 4317:4317 \ - -p 4318:4318 \ - grafana/tempo:2.6.1 + grafana/tempo:2.9.4 ``` +> Note that xrpld itself exports OTLP/**HTTP** only (§2.2.1), so it cannot send +> to this container directly — the collector is what bridges HTTP ingest to +> Tempo's gRPC receiver (`otlp/tempo` → `tempo:4317`). A bare Tempo container is +> useful for replaying traces from another OTLP/gRPC producer, not as an xrpld +> endpoint. + +> In practice, prefer the full stack — +> `docker compose -f docker/telemetry/docker-compose.yml up -d` — over a bare +> Tempo container. Most shipped dashboards query Prometheus span metrics, which +> need the collector and Prometheus services too. See +> [05 §5.6](./05-configuration-reference.md). + --- ## 7.2 Production Backends @@ -168,47 +187,100 @@ flowchart TB ### 7.4.2 Sampling Strategy +An earlier version of this section described a three-policy tail sampler (keep +all errors / keep anything >5s / keep 10% of the rest). **No such sampler +exists in this repo.** What ships is below. + ```mermaid flowchart LR - subgraph head["Head Sampling (Node)"] - hs[Node-level head sampling
fixed at 100%
not configurable] + subgraph head["Head Sampling (Node) — fixed"] + hs["ParentBased(TraceIdRatio 1.0)
samplingRatio is static constexpr
no config key exists
100% of spans exported"] end - subgraph tail["Tail Sampling (Collector)"] - ts1[Keep all errors] - ts2[Keep slow >5s] - ts3[Keep 10% rest] + subgraph tail["Tail Sampling (Collector) — opt-in"] + base["Base config:
NO tail_sampling processor
100% of traces stored"] + cloud["grafanacloud overlay only:
one probabilistic policy
sampling_percentage: 0.5"] end head --> tail + base --> final["Stored Traces"] + cloud --> final - ts1 --> final[Final Traces] - ts2 --> final - ts3 --> final - - style head fill:#0d47a1,stroke:#082f6a,color:#fff - style tail fill:#1b5e20,stroke:#0d3d14,color:#fff - style hs fill:#0d47a1,stroke:#082f6a,color:#fff - style ts1 fill:#1b5e20,stroke:#0d3d14,color:#fff - style ts2 fill:#1b5e20,stroke:#0d3d14,color:#fff - style ts3 fill:#1b5e20,stroke:#0d3d14,color:#fff - style final fill:#bf360c,stroke:#8c2809,color:#fff + style head fill:#0d47a1,stroke:#082f6a,color:#ffffff + style tail fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style hs fill:#0d47a1,stroke:#082f6a,color:#ffffff + style base fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style cloud fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style final fill:#bf360c,stroke:#8c2809,color:#ffffff ``` **Reading the diagram:** -- **Head Sampling (Node)**: xrpld pins head sampling at 100% (sample everything) and does not expose a configurable ratio. This is intentional: a per-node ratio would let different nodes make divergent keep/drop decisions for the same distributed trace, producing broken/partial traces. xrpld uses a `ParentBased` sampler so spans inheriting a remote parent honor the upstream decision. Volume reduction is delegated to the collector's tail sampling. -- **Tail Sampling (Collector)**: The second filter -- the collector inspects completed traces and applies rules: keep all errors, keep anything slower than 5 seconds, and keep 10% of the remainder. -- **Arrow head → tail**: All head-sampled traces flow to the collector, where tail sampling further reduces volume while preserving the most valuable data. -- **Final Traces**: The output after both sampling stages; this is what gets stored and queried. The two-stage approach balances cost with debuggability. +- **Head Sampling (Node)** — fixed at 100% and genuinely not configurable: + `Telemetry.h:234` declares `static constexpr double samplingRatio = 1.0;` and + `TelemetryConfig.cpp:139` records that there is nothing to parse. This is + intentional: a per-node ratio would let different nodes make divergent + keep/drop decisions for the same distributed trace, producing broken/partial + traces. The ratio sampler is wrapped in a `ParentBased` sampler so spans + inheriting a remote parent honour the upstream decision. +- **Tail Sampling (Collector)** — the base config + (`docker/telemetry/otel-collector-config.yaml`) has **no** `tail_sampling` + processor, so the local and CI stacks keep 100% of traces. The only shipped + policy lives in `otel-collector-config.grafanacloud.yaml:60-67`, wired into + the **`traces/store`** pipeline (`:259-261`) — the overlay has no pipeline + named `traces`; it splits the trace stream into `traces/metrics` (unsampled, + feeds `spanmetrics`) and `traces/store` (sampled, feeds Tempo and Grafana + Cloud). See [05 §5.5.2](./05-configuration-reference.md) for the full overlay + delta. The policy is a single `probabilistic` at **0.5%**, + `decision_wait: 10s`, `num_traces: 50000`. There are no error or latency + carve-outs. +- **Why 0.5% does not damage the dashboards**: the policy is applied on the + trace-storage branch only. The `spanmetrics` connector runs on a separate + branch that still sees every span, so `span_calls_total` and + `span_duration_milliseconds_*` remain exact. Sampling costs you individual + example traces in Tempo, not metric accuracy. +- **If you want the error/latency policies**: they are a reasonable thing to + add, but they must be written — and `decision_wait` sized so a trace's spans + have all arrived before the policy evaluates it. + +#### Companion guard: `memory_limiter` (recommended, not configured) + +Tail sampling bounds what the collector **stores**; it does not bound what the +collector **buffers**. `tail_sampling` is the opposite of cheap here — it holds +up to `num_traces` (50 000) traces in memory for `decision_wait` before +deciding — and the `spanmetrics` connector keeps a live series cache on top of +that. A production gateway collector should therefore also run a +[`memory_limiter`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md) +processor as an OOM guard: it applies backpressure (refusing new data with a +retryable error, which the node's `sending_queue` will retry) instead of letting +the process be killed and losing every buffered trace. + +> **Not currently configured anywhere in this repo.** Neither +> `otel-collector-config.yaml` nor +> `otel-collector-config.grafanacloud.yaml` declares a `memory_limiter`, and +> neither compose file sets a container memory limit — so today a traffic spike +> is bounded only by host RAM. This is a recommendation for real deployments, +> recorded here because [05 §5.5.1](./05-configuration-reference.md) lists +> `memory_limiter` among the processors deliberately **absent** from the shipped +> config and that must not be read as "not needed". Placement rules if you add +> it: it must be the **first** processor in every pipeline (ahead of `batch`), +> and `limit_mib` must sit below the container/cgroup limit with headroom for +> the sampling and spanmetrics caches. ### 7.4.3 Data Retention -| Environment | Hot Storage | Warm Storage | Cold Archive | -| ----------- | ----------- | ------------ | ------------ | -| Development | 24 hours | N/A | N/A | -| Staging | 7 days | N/A | N/A | -| Production | 7 days | 30 days | many years | +| Environment | Hot Storage | Warm Storage | Cold Archive | Source | +| --------------------------- | ----------- | ------------ | ------------ | ------------------------------------------------------------ | +| Development (local stack) | **1 hour** | N/A | N/A | `tempo.yaml:40` — `compactor.compaction.block_retention: 1h` | +| Staging (recommendation) | 7 days | N/A | N/A | Not configured in this repo | +| Production (recommendation) | 7 days | 30 days | many years | Not configured in this repo | + +> **The local stack keeps traces for 1 hour, not 24.** `block_retention: 1h` +> is deliberate — it bounds disk for a long-running dev node — but it means a +> trace you found this morning is gone by lunchtime. Raise +> `block_retention` in `docker/telemetry/tempo.yaml` before starting any +> investigation that needs to span a working day. The staging and production +> rows are recommendations only; nothing in this repo provisions them. --- @@ -224,56 +296,106 @@ flowchart LR --- -## 7.6 Grafana Dashboard Examples +## 7.6 Grafana Dashboards and Alerts -Pre-built dashboards for xrpld observability. +> **Superseded.** This section was written in Phase 1a, before any dashboard +> shipped, and described three hypothetical boards (`xrpld-consensus-health`, +> `xrpld-node-overview`, `xrpld-unified`) and three TraceQL alert rules in a +> group called `xrpld-tracing-alerts`. **None of those uids or rule names exist +> anywhere in the repo.** What actually ships is 15 dashboards and 13 alert +> rules, and both are Prometheus-first rather than TraceQL-first. The +> authoritative references are: +> +> | For | See | +> | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | +> | Dashboard and panel inventory, per-panel queries | [09-data-collection-reference.md](./09-data-collection-reference.md) | +> | Alert catalogue, thresholds and response steps | `docs/telemetry-runbook.md` | +> | Files on disk | `docker/telemetry/grafana/dashboards/*.json`, `docker/telemetry/grafana/provisioning/alerting/rules.yaml` | +> +> The rest of this section records only the facts a reader needs so as not to +> chase the removed names. -### 7.6.1 Consensus Health Dashboard +### 7.6.1 Shipped Dashboards -A Tempo-backed dashboard (uid `xrpld-consensus-health`) with four panels, all driven by TraceQL: +15 JSON dashboards are provisioned into Grafana folder `xrpld`. The uids are +bare — there is no `xrpld-` prefix: -- **Consensus Round Duration** (timeseries, ms): average `consensus.round` span duration per node instance, with yellow/red thresholds at 4s/5s. -- **Phase Duration Breakdown** (barchart): average duration of `consensus.phase.*` spans grouped by span name. -- **Proposers per Round** (stat): average of the `span.proposers` attribute on `consensus.round` spans. -- **Recent Slow Rounds (>5s)** (table): `consensus.round` spans filtered to `duration > 5s`. +`consensus-health`, `fee-market`, `job-queue`, `ledger-data-sync`, +`ledger-operations`, `log-derived-insights`, `network-traffic`, `node-health`, +`overlay-traffic-detail`, `peer-network`, `peer-quality`, `rpc-pathfinding`, +`rpc-performance`, `transaction-overview`, `validator-health`. -Each panel's TraceQL query is described inline in its bullet above. +> **Panel-count convention** (shared with [05 §5.8.3](./05-configuration-reference.md)): +> counts are of **data panels only**. `type: "row"` collapsible headers are +> excluded because a row carries no query, so a board's raw `panels` array is +> longer than its stated count. -### 7.6.2 Node Overview Dashboard +`consensus-health.json` is a useful calibration for how far this section drifted: +where the removed text described "four TraceQL panels", the real board carries **22 +data panels** in 4 rows (26 `panels` array entries) — 19 Prometheus targets +against `${DS_PROMETHEUS}` and 9 TraceQL targets against `${DS_TEMPO}`. Tempo is +used for trace _drill-down_; the time series come from span metrics. -A Tempo-backed dashboard (uid `xrpld-node-overview`) with four panels: +### 7.6.2 Shipped Alert Rules -- **Active Nodes** (stat): count of distinct `resource.service.instance.id` values seen for the `xrpld` service. -- **Total Transactions (1h)** (stat): count of `tx.receive` spans. -- **Error Rate** (gauge, percent): ratio of `status.code=error` spans to all spans, with yellow/red thresholds at 1%/5%. -- **Service Map** (nodeGraph): Tempo-generated service dependency graph. +`docker/telemetry/grafana/provisioning/alerting/rules.yaml` provisions **13 +rules in 5 groups**, all in folder `xrpld`, all `interval: 1m`, and all +**PromQL** — there are zero TraceQL alert rules. -### 7.6.3 Alert Rules +| Group | Rules | +| ------------------ | --------------------------------------------------------------------------- | +| `xrpld-consensus` | `LedgerHistoryMismatch`, `LedgerCloseStalled`, `ValidatedLedgerStale` | +| `xrpld-validator` | `ValidationsMissed`, `ValidationsNotChecked` | +| `xrpld-jobqueue` | `JobQueueTxOverflow`, `JobQueueLatencyHigh`, `NodeStoreIOLatencyHigh` | +| `xrpld-node-state` | `NodeStateFlapping`, `NodeNotFull` | +| `xrpld-overlay` | `ManifestJobQueueConvoy`, `ManifestFloodInbound`, `PeerResourceDisconnects` | -Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`, evaluated every 1m) against the Tempo datasource: +> Two placements are worth noting because they are not what the rule name +> suggests. `ValidatedLedgerStale` is grouped under `xrpld-consensus`, not +> `xrpld-validator` — it fires on any node whose validated-ledger sequence stops +> advancing, which is a chain-progress symptom rather than a validator-identity +> one. `NodeStoreIOLatencyHigh` is grouped under `xrpld-jobqueue`, not +> `xrpld-node-state` — slow NodeStore I/O manifests first as job-queue backlog, +> so grouping it there keeps the cause and its effect in one notification. -- **Consensus Round Slow** (warning, `for: 5m`): fires when average `consensus.round` duration exceeds 5s. +Thresholds, measured baselines and response procedures are in the runbook's +alert catalogue, not here. - ``` - {resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s - ``` +### 7.6.3 Writing New Rules: the metric name -- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. Error _rate_ is a ratio, so it must divide the error-span rate by the total-span rate — a single TraceQL `rate()` returns spans/second, not a percentage, and would fire on traffic volume alone. This uses span metrics emitted by the collector's `spanmetrics` connector (Prometheus datasource), not a TraceQL query: +If you add a span-metric alert, the metric is **`span_calls_total`**. This stack +sets the `spanmetrics` connector's `namespace: "span"` +(`otel-collector-config.yaml:114`); the connector's own default namespace is +**empty**, so without that setting the names would be the bare `calls_total` / +`duration_milliseconds_*`. 7 of the 15 dashboards already query the `span_` +names. Durations are likewise `span_duration_milliseconds_bucket`. - ``` - sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m])) - / - sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m])) - > 0.05 - ``` +> **`traces_spanmetrics_*` is a different producer, not the connector's +> default.** That family is emitted by **Tempo's** `metrics_generator` +> `span-metrics` processor (`tempo.yaml:70-76`), which is a separate +> implementation from the collector connector. It does not exist in this stack +> either: the generator's `remote_write` is commented out (`tempo.yaml:53-56`) +> and `prometheus.yml:6-9` scrapes only `otel-collector:8889`, so nothing stores +> what Tempo generates. Do not write a rule against `traces_spanmetrics_*` and +> do not describe `namespace: "span"` as overriding it. -- **Transaction Throughput Drop** (warning, `for: 10m`): fires when the `tx.receive` span rate falls below 10/s. +An RPC error-rate rule, written against the real metric name, looks like this. +Note that error _rate_ is a ratio, so it must divide the error-span rate by the +total-span rate — a bare rate returns calls/second and would fire on traffic +volume alone: - ``` - {resource.service.name="xrpld" && name="tx.receive"} | rate() < 10 - ``` +``` +sum(rate(span_calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m])) +/ +sum(rate(span_calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m])) +> 0.05 +``` -> **Note**: The Consensus Round Slow and Transaction Throughput Drop rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning. The RPC Error Rate Spike rule instead queries Prometheus span metrics (collector `spanmetrics` connector), so it needs that connector enabled in the collector pipeline. +> **Prefer PromQL over TraceQL for alerting.** TraceQL aggregates +> (`avg(duration)`, `rate()`) need Tempo 2.3+ with TraceQL metrics enabled, are +> slower, and are distorted by any tail sampling in the path (§7.4.2). Span +> metrics are computed pre-sampling and cost nothing extra to query. That is +> why all 13 shipped rules are PromQL. --- @@ -285,81 +407,73 @@ How to correlate OpenTelemetry traces with existing xrpld observability. ### 7.7.1 Correlation Architecture +There is **one** collection agent, not three. Earlier drafts of this diagram +routed logs through "Promtail/Fluentd" and metrics through a "StatsD Exporter"; +neither exists in this stack. Logs are read by the OTel Collector's own +`filelog` receiver, and `beast::insight` metrics arrive at the same collector +over OTLP (`[insight] server=otel`). The single-agent shape is the point: one +process, one config file, one place to add redaction or tier tagging. + ```mermaid flowchart TB subgraph xrpld["xrpld Node"] - otel[OpenTelemetry
Spans] - perflog[PerfLog
JSON Logs] - insight[Beast Insight
StatsD Metrics] + otel["OpenTelemetry Spans"] + journal["Journal debug.log
trace_id= span_id= prefix
(Log.cpp:304-338)"] + insight["Beast Insight + XRPL_METRIC_*
native OTLP metrics"] end - subgraph collectors["Data Collection"] - otelc[OTel Collector] - promtail[Promtail/Fluentd] - statsd[StatsD Exporter] - end + otelc["OTel Collector
receivers: otlp, filelog
connector: spanmetrics
3 pipelines"] subgraph storage["Storage"] - tempo[(Tempo)] - loki[(Loki)] - prom[(Prometheus)] + tempo[("Tempo")] + loki[("Loki")] + prom[("Prometheus")] end - subgraph grafana["Grafana"] - traces[Trace View] - logs[Log View] - metrics[Metrics View] - corr[Correlation
Panel] - end + dashboards["Grafana
Tempo to Loki via tracesToLogs
Loki to Tempo via derived fields"] - otel -->|OTLP| otelc --> tempo - perflog -->|JSON| promtail --> loki - insight -->|StatsD| statsd --> prom + otel -->|"OTLP/HTTP :4318"| otelc + journal -->|"filelog tails
/var/log/xrpld"| otelc + insight -->|"OTLP/HTTP :4318"| otelc - tempo --> traces - loki --> logs - prom --> metrics + otelc -->|"otlp/tempo"| tempo + otelc -->|"otlphttp/loki"| loki + otelc -->|"prometheus :8889"| prom - traces --> corr - logs --> corr - metrics --> corr + tempo --> dashboards + loki --> dashboards + prom --> dashboards - style xrpld fill:#0d47a1,stroke:#082f6a,color:#fff - style collectors fill:#bf360c,stroke:#8c2809,color:#fff - style storage fill:#1b5e20,stroke:#0d3d14,color:#fff - style grafana fill:#4a148c,stroke:#2e0d57,color:#fff - style otel fill:#0d47a1,stroke:#082f6a,color:#fff - style perflog fill:#0d47a1,stroke:#082f6a,color:#fff - style insight fill:#0d47a1,stroke:#082f6a,color:#fff - style otelc fill:#bf360c,stroke:#8c2809,color:#fff - style promtail fill:#bf360c,stroke:#8c2809,color:#fff - style statsd fill:#bf360c,stroke:#8c2809,color:#fff - style tempo fill:#1b5e20,stroke:#0d3d14,color:#fff - style loki fill:#1b5e20,stroke:#0d3d14,color:#fff - style prom fill:#1b5e20,stroke:#0d3d14,color:#fff - style traces fill:#4a148c,stroke:#2e0d57,color:#fff - style logs fill:#4a148c,stroke:#2e0d57,color:#fff - style metrics fill:#4a148c,stroke:#2e0d57,color:#fff - style corr fill:#4a148c,stroke:#2e0d57,color:#fff + style xrpld fill:#0d47a1,stroke:#082f6a,color:#ffffff + style storage fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style otel fill:#0d47a1,stroke:#082f6a,color:#ffffff + style journal fill:#0d47a1,stroke:#082f6a,color:#ffffff + style insight fill:#0d47a1,stroke:#082f6a,color:#ffffff + style otelc fill:#bf360c,stroke:#8c2809,color:#ffffff + style tempo fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style loki fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style prom fill:#1b5e20,stroke:#0d3d14,color:#ffffff + style dashboards fill:#4a148c,stroke:#2e0d57,color:#ffffff ``` **Reading the diagram:** -- **xrpld Node (three sources)**: A single node emits three independent data streams -- OpenTelemetry spans, PerfLog JSON logs, and Beast Insight StatsD metrics. -- **Data Collection layer**: Each stream has its own collector -- OTel Collector for spans, Promtail/Fluentd for logs, and a StatsD exporter for metrics. They operate independently. -- **Storage layer (Tempo, Loki, Prometheus)**: Each data type lands in a purpose-built store optimized for its query patterns (trace search, log grep, metric aggregation). -- **Grafana Correlation Panel**: The key integration point -- Grafana queries all three stores and links them via shared fields (`trace_id`, `tx_hash`, `ledger_seq`), enabling a single-pane debugging experience. +- **xrpld Node (three signals, one transport)**: spans and metrics both leave over OTLP/HTTP on port 4318. Logs do not leave the node at all — the node just writes `debug.log`, and the journal sink prefixes `trace_id=`/`span_id=` whenever a span is active (`Log.cpp:304-338`). +- **OTel Collector (single agent)**: an `otlp` receiver takes spans and metrics; a `filelog` receiver tails `/var/log/xrpld/*/debug.log` and regex-parses the trace/span IDs out of each line. A `spanmetrics` connector derives RED metrics from the trace stream and feeds them into the metrics pipeline. Three pipelines, three exporters — see [05 §5.5.1](./05-configuration-reference.md). +- **PerfLog is not in this picture.** It still writes `perf.log`, but nothing collects it and it carries no trace ID; the `setTraceId` hook once planned for it was never built ([02 §2.6.5](./02-design-decisions.md)). +- **StatsD is not in this picture either.** It remains a supported `[insight] server=` choice, but selecting it takes metrics _out_ of this pipeline and requires a StatsD receiver you would have to add yourself — the compose file's StatsD port mapping is commented out. +- **Grafana**: correlation is bidirectional and configured in the datasources, not in a bespoke panel — Tempo's `tracesToLogs` (`filterByTraceID: true`) jumps trace → logs, and `loki.yaml`'s derived fields jump log → trace. ### 7.7.2 Correlation Fields -| Source | Field | Link To | Purpose | -| ----------- | ------------------- | ------------- | -------------------------- | -| **Trace** | `trace_id` | Logs | Find log entries for trace | -| **Trace** | `tx_hash` | Logs, Metrics | Find TX-related data | -| **Trace** | `ledger_seq` | Logs | Find ledger-related logs | -| **PerfLog** | `trace_id` (new) | Traces | Jump to trace from log | -| **PerfLog** | `ledger_seq` | Traces | Find consensus trace | -| **Insight** | `exemplar.trace_id` | Traces | Jump from metric spike | +| Source | Field | Link To | Status | +| --------------- | --------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Trace** | `trace_id` | Logs | **Live.** Tempo `tracesToLogs`, `filterByTraceID: true` | +| **Trace** | `tx_hash` | — | Live as a span attribute for search; **not** used as a cross-signal join key (`tags: []`) | +| **Trace** | `ledger_seq` | — | Live as a span attribute; not a join key | +| **Journal log** | `trace_id`, `span_id` | Traces | **Live.** Emitted by `Log.cpp:304-338` into `debug.log`, parsed by the collector's `filelog` receiver, jumped via `loki.yaml` derived fields | +| **PerfLog** | `trace_id` | Traces | **Not implemented.** PerfLog output has no trace ID; the planned `setTraceId` hook was never built. Use the journal log instead | +| **Insight** | `exemplar.trace_id` | Traces | **Not implemented.** No exemplar configuration exists anywhere in the code or collector config — no `exemplar_filter` on the SDK side, no `exemplarTraceIdDestinations` on the Prometheus datasource. Metric spike → trace jumps must be done by time range today | ### 7.7.3 Example: Debugging a Slow Transaction @@ -376,31 +490,82 @@ flowchart TB Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 ``` -**Step 3: Find related PerfLog entries** +**Step 3: Find related log lines** ``` -# In Grafana Explore with Loki -{job="xrpld"} |= "4bf92f3577b34da6a3ce929d0e0e4736" +# In Grafana Explore with Loki. `service_name` is the promoted stream label; +# do NOT use {job="xrpld"} — see the note below. +{service_name="xrpld"} |= "4bf92f3577b34da6a3ce929d0e0e4736" ``` -**Step 4: Check Insight metrics for the time window** +These are journal (`debug.log`) lines, not PerfLog lines — see §7.7.2. + +> **Known issue — `{job="xrpld"}` does not select anything.** The collector's +> `resource/logs` processor does upsert a `job=xrpld` resource attribute +> (`otel-collector-config.yaml:62-70`), explicitly so that operators could paste +> `{job="xrpld"}`. Loki does not cooperate: on OTLP ingest it promotes only an +> **allow-listed** set of resource attributes to indexed stream labels +> (`service.name`, `service.namespace`, `service.instance.id`, +> `deployment.environment`, `k8s.*`, `cloud.*`), and `job` is not on it. This +> repo mounts no Loki config override (`docker-compose.yml:75` uses the image's +> built-in `local-config.yaml`), so `job` lands in **structured metadata** — +> queryable only with a `|` filter after a selector, never as the selector +> itself. A `{job="xrpld"}` query returns empty with no error, which is why this +> is easy to miss. `docs/telemetry-runbook.md:2533` says the same, and all 38 +> Loki queries in the shipped dashboards (35 panel targets + 3 template +> variables) select on `service_name` — zero use `job`. Fix options: +> drop the ineffective `job` upsert, or mount a Loki config adding `job` to +> `distributor.otlp_config.resource_attributes`. + +**Step 4: Check metrics for the time window** ``` -# In Grafana with Prometheus -rate(xrpld_tx_applied_total[1m]) - @ timestamp_from_trace +# In Grafana with Prometheus. Span-derived RED metrics for the transaction +# pipeline (namespace "span" — see 7.6.3): +sum(rate(span_calls_total{span_name="tx.process"}[1m])) by (service_instance_id) + +# Error share of the same pipeline. Note !~"tesSUCCESS|" — NOT +# !="tesSUCCESS" — so spans that carry no ter_result are excluded: +sum(rate(span_calls_total{span_name="tx.process", ter_result!~"tesSUCCESS|"}[5m])) +/ +sum(rate(span_calls_total{span_name="tx.process"}[5m])) ``` -### 7.7.4 Unified Dashboard Example +> **Why the regex form.** An absent Prometheus label is indistinguishable from +> the empty string, and `tx.process` can end **without** a `ter_result`: the span +> is opened at `NetworkOPs.cpp:1416`, but `processTransaction()` returns early +> when `preProcessTransaction()` rejects the transaction (`:1437-1438`), and +> `doTransactionAsync()` returns early when the transaction is already applying +> (`:1461-1462`) — both before the only setter, at `:1674`. Those series arrive +> with `ter_result=""`, which `!="tesSUCCESS"` happily counts as a failure and +> inflates the ratio. `!~"tesSUCCESS|"` excludes the empty value via the trailing +> `|` alternative. This is the form `docs/telemetry-runbook.md:1198` and the +> `transaction-overview.json` stage-failure panels already use; apply it to any +> new `ter_result` predicate. -A single dashboard (uid `xrpld-unified`) that ties traces, metrics, and logs together across the Tempo, Prometheus, and Loki datasources: +> Earlier drafts used `rate(xrpld_tx_applied_total[1m])` and +> `rate(xrpld_tx_received_total[5m])`. **Neither metric exists** — there is no +> `xrpld_`-prefixed metric family at all, because `OTelCollector::formatName()` +> deliberately prepends no prefix (`OTelCollector.cpp:855-866`); the OTel +> resource `service.name` identifies the service instead. Use the `span_*` +> families above (verified in `transaction-overview.json` and +> `rpc-performance.json`) or the native `XRPL_METRIC_*` instrument names listed +> in [09-data-collection-reference.md](./09-data-collection-reference.md). -- **Transaction Latency (Traces)** (timeseries, Tempo): `histogram_over_time(duration)` of `tx.receive` spans. -- **Transaction Rate (Metrics)** (timeseries, Prometheus): `rate(xrpld_tx_received_total[5m])` per instance, with a data link that opens the matching `tx.receive` traces in Tempo. -- **Recent Logs** (logs, Loki): `{job="xrpld"} | json`. -- **Trace Search** (table, Tempo): all `xrpld` traces, with per-row data links on `traceID` that jump to the trace in Tempo and to the correlated logs in Loki (`{job="xrpld"} |= ""`). +### 7.7.4 Unified Dashboard -The cross-datasource data links are what make this a single-pane debugging view; the correlation fields they rely on are listed in section 7.7.2. +> **Superseded.** No `xrpld-unified` dashboard exists. The single-pane view it +> described is instead delivered by two things that did ship: the +> **`log-derived-insights`** dashboard (31 data panels in 10 rows, all +> Loki-backed — 41 `panels` array entries; see the counting convention in +> §7.6.1) plus the +> bidirectional datasource links (Tempo `tracesToLogs` → Loki, `loki.yaml` +> derived fields → Tempo), which let you cross signals from _any_ board rather +> than only from one dedicated dashboard. +> +> The correlation fields those links rely on — and which of them are actually +> implemented — are in §7.7.2. For the full board inventory see +> [09-data-collection-reference.md](./09-data-collection-reference.md). --- diff --git a/OpenTelemetryPlan/08-appendix.md b/OpenTelemetryPlan/08-appendix.md index c75af13d12..3cff77a23f 100644 --- a/OpenTelemetryPlan/08-appendix.md +++ b/OpenTelemetryPlan/08-appendix.md @@ -133,18 +133,25 @@ The full span inventory (names, attributes, parents as instrumented) is in ### Task Lists -| Document | Description | -| -------------------------------------------------------------------------- | --------------------------------------------------- | -| [Phase2_taskList.md](./Phase2_taskList.md) | RPC layer trace instrumentation | -| [Phase3_taskList.md](./Phase3_taskList.md) | Peer overlay & consensus tracing | -| [Phase4_taskList.md](./Phase4_taskList.md) | Transaction lifecycle tracing | -| [Phase5_taskList.md](./Phase5_taskList.md) | Ledger processing & advanced tracing | -| [Phase5_IntegrationTest_taskList.md](./Phase5_IntegrationTest_taskList.md) | Observability stack integration tests | -| [Phase7_taskList.md](./Phase7_taskList.md) | Native OTel metrics migration | -| [Phase8_taskList.md](./Phase8_taskList.md) | Log-trace correlation | -| [Phase9_taskList.md](./Phase9_taskList.md) | Internal metric instrumentation gap fill (future) | -| [Phase10_taskList.md](./Phase10_taskList.md) | Synthetic workload generation & validation (future) | -| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) | +| Document | Description | +| -------------------------------------------------------------------------- | ---------------------------------------------- | +| [Phase2_taskList.md](./Phase2_taskList.md) | RPC layer trace instrumentation | +| [Phase3_taskList.md](./Phase3_taskList.md) | Peer overlay & consensus tracing | +| [Phase4_taskList.md](./Phase4_taskList.md) | Transaction lifecycle tracing | +| [Phase5_taskList.md](./Phase5_taskList.md) | Ledger processing & advanced tracing | +| [Phase5_IntegrationTest_taskList.md](./Phase5_IntegrationTest_taskList.md) | Observability stack integration tests | +| [Phase7_taskList.md](./Phase7_taskList.md) | Native OTel metrics migration | +| [Phase8_taskList.md](./Phase8_taskList.md) | Log-trace correlation | +| [Phase9_taskList.md](./Phase9_taskList.md) | Internal metric instrumentation gap fill | +| [Phase10_taskList.md](./Phase10_taskList.md) | Synthetic workload generation & validation | +| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) | + +> **Only Phase 11 is still "future".** Phase 9 ships on +> `pratik/otel-phase9-metric-gap-fill` (18 task entries, 9.1–9.17 plus 9.7a) and +> Phase 10 on `pratik/otel-phase10-workload-validation` (7 tasks). Their task +> lists are present on every branch from those points forward, so a reader on a +> later branch sees plans that are already implemented, not proposals. Phase 11 +> (13 tasks) has no implementation branch. > **Note**: Phases 1 and 6 do not have separate task list files. Phase 1 tasks are documented in [06-implementation-phases.md §6.2](./06-implementation-phases.md). Phase 6 tasks are documented in [06-implementation-phases.md §6.7](./06-implementation-phases.md). @@ -156,13 +163,21 @@ This guide maps Phase 9–11 content to its location across the documentation. ### Phase 9: Internal Metric Instrumentation Gap Fill -| Content | Location | -| ------------------------------- | ------------------------------------------------------------------------ | -| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | -| Task list (10 tasks) | [Phase9_taskList.md](./Phase9_taskList.md) | -| Future metric definitions (~50) | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | -| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (planned) | -| New dashboards | `fee-market`, `job-queue` (planned) | +| Content | Location | +| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | +| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) | +| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | +| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` — **shipped** | +| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** | +| Updated dashboards (2) | `node-health`, `rpc-performance` | +| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) | + +> **Task numbering**: `Phase9_taskList.md` carries 18 `## Task 9.x` headings — +> 9.1 through 9.17 plus the inserted 9.7a (`push_metrics.py` parity). The "10 +> tasks" figure in earlier revisions predates 9.7a and 9.11–9.17. Tasks 9.8 and +> 9.11–9.13 together produce the four new dashboards; Task 9.17 (peer span +> coverage) is explicitly **deferred to Phase 11**. **Metric categories**: NodeStore I/O, Cache Hit Rates, TxQ, PerfLog Per-RPC, PerfLog Per-Job, Counted Objects, Fee Escalation & Load Factors. @@ -172,23 +187,45 @@ This guide maps Phase 9–11 content to its location across the documentation. | -------------------- | ------------------------------------------------------------------------ | | Plan & architecture | [06-implementation-phases.md §6.8.3](./06-implementation-phases.md) | | Task list (7 tasks) | [Phase10_taskList.md](./Phase10_taskList.md) | +| Branch | `pratik/otel-phase10-workload-validation` | | Validation inventory | [09-data-collection-reference.md §5c](./09-data-collection-reference.md) | -| Test harness | `docker/telemetry/docker-compose.workload.yaml` (planned) | -| CI workflow | `.github/workflows/telemetry-validation.yml` (planned) | +| Test harness | `docker/telemetry/docker-compose.workload.yaml` (phase-10 branch) | +| CI workflow | `.github/workflows/telemetry-validation.yml` (phase-10 branch) | -**Validates**: 16 spans, 22 attributes, 300+ metrics, 10 dashboards, log-trace correlation. +**Validates** (Phase-10 harness inventory): **40** span types, **67** unique +required span attributes, **36** metric entries, **14** dashboards, log-trace +correlation. + +> **These are the harness manifests' counts, and two of them lag the code.** The +> manifests (`docker/telemetry/workload/expected_spans.json`, +> `expected_metrics.json`) live only on the phase-10 branch. `expected_spans.json` +> holds 40 span entries against the **41** span-name families the code emits +> (`rpc.ws_upgrade` has no entry), and its own `total_unique_attributes: 58` field +> is stale against the 67 attributes its per-span `required_attributes` lists +> actually name. `expected_metrics.json` asserts 14 dashboard uids against the +> **15** dashboard JSONs in `docker/telemetry/grafana/dashboards/`; +> `log-derived-insights` is the unasserted one. The full emitted inventory is in +> [09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-41-spans) +> and [§5c](./09-data-collection-reference.md#validated-telemetry-inventory). ### Phase 11: Third-Party Data Collection Pipelines | Content | Location | | --------------------------------- | ------------------------------------------------------------------------ | | Plan & architecture | [06-implementation-phases.md §6.8.4](./06-implementation-phases.md) | -| Task list (11 tasks) | [Phase11_taskList.md](./Phase11_taskList.md) | +| Task list (13 tasks) | [Phase11_taskList.md](./Phase11_taskList.md) | | External metric definitions (~30) | [09-data-collection-reference.md §5d](./09-data-collection-reference.md) | | Custom OTel Collector receiver | `docker/telemetry/otel-rippled-receiver/` (planned) | | Prometheus alerting rules (11) | [09-data-collection-reference.md §5d](./09-data-collection-reference.md) | | New dashboards (4) | Validator Health, Network Topology, Fee Market (External), DEX & AMM | +> **Two of those names now collide with shipped Phase-9 boards.** Phase 9 +> already ships `validator-health` and `fee-market`, both built from the node's +> **own** telemetry. The Phase-11 entries are the third-party-data variants +> (network-wide validator agreement, external fee/DEX feeds via the custom +> receiver). They need distinct uids, or they will overwrite the Phase-9 boards +> on provisioning. + **Consumer categories**: Exchanges, Payment Processors, DeFi/AMM, NFT Marketplaces, Analytics Providers, Wallets, Compliance, Academic Researchers, Institutional Custody, CBDC Bridge Operators. --- diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index aec3e9dd40..bee2152837 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -2,7 +2,7 @@ > **Audience**: Developers and operators. This is the single source of truth for all telemetry data collected by xrpld's observability stack. > -> **Related docs**: [docs/telemetry-runbook.md](../docs/telemetry-runbook.md) (operator runbook with alerting and troubleshooting) | [03-implementation-strategy.md](./03-implementation-strategy.md) (code structure and performance optimization) | [04-code-samples.md](./04-code-samples.md) (C++ instrumentation examples) +> **Related docs**: [docs/telemetry-runbook.md](../docs/telemetry-runbook.md) (operator runbook with alerting and troubleshooting) | [03-implementation-strategy.md](./03-implementation-strategy.md) (code structure and performance optimization) | [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) (authoritative span-flow reference; replaces the deleted `04-code-samples.md`) ## Data Flow Overview @@ -33,7 +33,7 @@ graph LR end subgraph viz["Visualization"] - F["Grafana :3000
15 dashboards"] + F["Grafana :3000
16 dashboards"] end A -->|"OTLP/HTTP :4318
(traces + attributes)"| R1 @@ -72,9 +72,13 @@ There are three independent telemetry pipelines entering a single **OTel Collect A third, narrower metrics path exists for instruments created at their call site through the `XRPL_METRIC_*` macros. These use the OTel Metrics SDK directly and reach the collector's OTLP -receiver rather than the StatsD receiver, so their names carry no `xrpld_` prefix. See -[§2a](#2a-call-site-otel-metrics-metricsregistry). Code in `libxrpl` cannot use these macros and -always goes through `beast::insight` instead. +receiver rather than the StatsD receiver, so their names carry no `xrpld_` prefix. The seven +call-site instruments are documented with the families they belong to: +`rpc_in_flight_requests` in +[§Per-RPC Method Metrics](#per-rpc-method-metrics-synchronous-countershistogram), the five +`getobject_*` in [§GetObject Request Path](#getobject-request-path-synchronous-countershistograms), +and `ledgers_closed_total` in [§Synchronous Counters (Phase 7+)](#synchronous-counters-phase-7). +Code in `libxrpl` cannot use these macros and always goes through `beast::insight` instead. **Trace backend** — The collector exports traces via OTLP/gRPC to: @@ -86,13 +90,22 @@ always goes through `beast::insight` instead. ## 1. OpenTelemetry Spans -### 1.1 Complete Span Inventory (~37 spans) +### 1.1 Complete Span Inventory (41 spans) -> **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [04-code-samples.md §4.6](./04-code-samples.md#46-span-flow-visualization) for span flow diagrams. +> **41 emitted span-name families.** The count is derived from the `*SpanNames.h` +> headers and their call sites, one family per distinct span name +> (`rpc.command.` and `grpc.` each count once, since the +> command / method name is a parameter of a single family). The tables below list +> all 41: RPC 5, gRPC 1, transaction 6, TxQ 6, consensus 13, ledger 4, peer 2, +> pathfind 4. The Phase-10 validation harness +> (`docker/telemetry/workload/expected_spans.json`) catalogues **40** of them — +> `rpc.ws_upgrade` has no entry. + +> **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) for the span flow diagrams (the former `04-code-samples.md` §4.6 was deleted). > **Span names vs. attribute keys**: span names use dotted `subsystem.operation` > form (e.g. `rpc.http_request`). Span _attribute_ keys use the bare/underscore -> form from the 2026-05-13 naming redesign (e.g. `tx_hash`, not `xrpl.tx.hash`). +> form from the 2026-05-13 naming redesign (e.g. `tx_hash`, not `xrpl.tx.hash`). > The dotted `xrpl.*` form is reserved for OTel **resource** attributes set once > at startup. See §1.2 for the full attribute inventory. @@ -190,26 +203,35 @@ Controlled by `trace_transactions=1` in `[telemetry]` config. Controlled by `trace_consensus=1` in `[telemetry]` config. -| Span Name | Parent | Source File | Description | -| ------------------------------ | ------------------ | ---------------- | ------------------------------------------------------------------- | -| `consensus.round` | — (root) | RCLConsensus.cpp | Root span for one consensus round (deterministic trace per round) | -| `consensus.phase.open` | `consensus.round` | Consensus.h | Open phase — collecting transactions before close | -| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal | -| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus | -| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — converging on the transaction set | -| `consensus.update_positions` | `consensus.round` | Consensus.h | Position update with per-dispute vote details | -| `consensus.check` | `consensus.round` | Consensus.h | Consensus threshold check (agree/disagree tally) | -| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) | -| `consensus.accept.apply` | `consensus.accept` | RCLConsensus.cpp | Ledger application with close-time details (jtACCEPT thread) | -| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted (follows-from link) | -| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Operating-mode transition during the round | -| `consensus.proposal.receive` | (context) | PeerImp.cpp | Proposal received from a peer (context-propagated into the round) | -| `consensus.validation.receive` | (context) | PeerImp.cpp | Validation received from a peer (context-propagated into the round) | +| Span Name | Parent | Source File | Description | +| ------------------------------ | --------------------- | ---------------- | ------------------------------------------------------------------- | +| `consensus.round` | — (root) | RCLConsensus.cpp | Root span for one consensus round (deterministic trace per round) | +| `consensus.phase.open` | `consensus.round` | Consensus.h | Open phase — collecting transactions before close | +| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal | +| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus | +| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — converging on the transaction set | +| `consensus.update_positions` | `consensus.establish` | Consensus.h | Position update with per-dispute vote details | +| `consensus.check` | `consensus.establish` | Consensus.h | Consensus threshold check (agree/disagree tally) | +| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) | +| `consensus.accept.apply` | `consensus.accept` | RCLConsensus.cpp | Ledger application with close-time details (jtACCEPT thread) | +| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted (follows-from link) | +| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Operating-mode transition during the round | +| `consensus.proposal.receive` | (context) | PeerImp.cpp | Proposal received from a peer (context-propagated into the round) | +| `consensus.validation.receive` | (context) | PeerImp.cpp | Validation received from a peer (context-propagated into the round) | The `.receive` spans are created per-message in the overlay and joined to the round trace via context propagation rather than direct parenting. The `consensus.validation.send` span uses a follows-from link off the round. +> **`update_positions` and `check` sit one level below `establish`, not below +> the round.** Both are created with +> `SpanGuard::childSpan(..., establishSpanContext_)` +> (`include/xrpl/consensus/Consensus.h:1628` and `:1837`), and +> `consensus.establish` is itself parented to `roundSpanContext_` +> (`Consensus.h:2099-2101`). An earlier revision of this table showed them as +> direct children of `consensus.round`; queries or trace-shape assertions built +> on that tree are wrong by one level. + **Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"consensus.*"}` **Grafana dashboard**: _Consensus Health_ (`consensus-health`) @@ -251,12 +273,12 @@ under an unrelated transaction's trace. Controlled by `trace_rpc=1` in `[telemetry]` config. -| Span Name | Parent | Source File | Description | -| --------------------- | -------------------- | --------------- | ---------------------------------------------------------- | -| `pathfind.request` | `rpc.command.` | PathFind.cpp | `path_find` RPC entry (`doPathFind`) | -| `pathfind.compute` | `pathfind.request` | PathRequest.cpp | Path computation for one request (`PathRequest::doUpdate`) | -| `pathfind.discover` | `pathfind.compute` | Pathfinder.cpp | Graph exploration (one per RPC call) | -| `pathfind.update_all` | — | PathRequest.cpp | Async recomputation of all active requests at ledger close | +| Span Name | Parent | Source File | Description | +| --------------------- | -------------------- | -------------------------------------- | ---------------------------------------------------------- | +| `pathfind.request` | `rpc.command.` | PathFind.cpp:27, RipplePathFind.cpp:36 | `path_find` / `ripple_path_find` RPC entry | +| `pathfind.compute` | `pathfind.request` | PathRequest.cpp:750 | Path computation for one request (`PathRequest::doUpdate`) | +| `pathfind.discover` | `pathfind.compute` | PathRequest.cpp:599-600 | Graph exploration (one per RPC call) | +| `pathfind.update_all` | — | PathRequestManager.cpp:88-92 | Async recomputation of all active requests at ledger close | > **Note**: `pathfind.request` nests under the active `rpc.command.` span. > Because OTel context storage is coroutine-aware (backed by `LocalValue`), the @@ -290,6 +312,8 @@ aggregation. Per the 2026-05-13 naming redesign, span-attribute keys use the > it (both `consensus.validation.send` and `peer.validation.receive`) — there > is no dotted span attribute. +The tables below list one row per attribute per subsystem, so a key shared by two subsystems (for example `ledger_seq`) appears once in each. That is 89 rows over 78 distinct keys. The §6 per-header counts use the same row-based rule, so they sum to 89. + #### RPC Attributes | Attribute | Type | Set On | Description | @@ -367,50 +391,78 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=}`. #### Consensus Attributes -| Attribute | Type | Set On | Description | -| -------------------------- | ------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | -| `consensus_ledger_id` | string | `consensus.round` | Previous-ledger id anchoring the round | -| `ledger_seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept.apply`, `consensus.validation.send` | Ledger sequence number | -| `consensus_mode` | string | `consensus.round`, `consensus.ledger_close` | Node mode: `"Proposing"`, `"Observing"`, `"Wrong"`, etc. | -| `consensus_round_id` | int64 | `consensus.round` | Round identifier | -| `consensus_phase` | string | `consensus.round` | Current phase name (updated on each transition) | -| `trace_strategy` | string | `consensus.round` | Trace-id strategy (`deterministic` / `random`) | -| `previous_ledger_seq` | int64 | `consensus.round` | Sequence of the previous ledger | -| `previous_proposers` | int64 | `consensus.round` | Proposer count in the previous round | -| `previous_round_time_ms` | int64 | `consensus.round` | Duration of the previous round | -| `consensus_round` | int64 | `consensus.proposal.send` | Proposal sequence number for the broadcast proposal | -| `is_bow_out` | boolean | `consensus.proposal.send` | Whether the proposal is a bow-out (resigning the round) | -| `tx_count_open` | int64 | `consensus.ledger_close` | Transactions in the open ledger at close | -| `close_time_resolution_ms` | int64 | `consensus.ledger_close` | Close-time rounding granularity | -| `converge_percent` | int64 | `consensus.establish`, `consensus.update_positions` | Convergence percentage | -| `establish_count` | int64 | `consensus.establish` | Establish-phase iteration count | -| `proposers` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.accept` | Number of proposers | -| `disputes_count` | int64 | `consensus.establish`, `consensus.update_positions` | Number of disputed transactions | -| `tx_id` | string | `consensus.update_positions` | Disputed transaction id (per-dispute event) | -| `dispute_our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed tx | -| `dispute_yays` | int64 | `consensus.update_positions` | Yes votes on the disputed tx | -| `dispute_nays` | int64 | `consensus.update_positions` | No votes on the disputed tx | -| `agree_count` | int64 | `consensus.check` | Agreeing proposer count | -| `disagree_count` | int64 | `consensus.check` | Disagreeing proposer count | -| `threshold_percent` | int64 | `consensus.check` | Agreement threshold percentage | -| `consensus_result` | string | `consensus.check` | Check outcome | -| `quorum` | int64 | `consensus.check`, `consensus.accept` | Quorum required | -| `round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds | -| `consensus_state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` | -| `close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) | -| `close_time_correct` | boolean | `consensus.accept.apply` | Whether validators agreed on close time | -| `close_resolution_ms` | int64 | `consensus.accept.apply` | Close-time rounding granularity in milliseconds | -| `proposing` | boolean | `consensus.accept.apply`, `consensus.validation.send` | Whether this node was a proposer | -| `parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger close time | -| `close_time_self` | int64 | `consensus.accept.apply` | This node's close-time vote | -| `close_time_vote_bins` | string | `consensus.accept.apply` | Distribution of close-time votes | -| `resolution_direction` | string | `consensus.accept.apply` | Whether close resolution increased/decreased/unchanged | -| `tx_count` | int64 | `consensus.accept.apply` | Transactions in the accepted set | -| `ledger_hash` | string | `consensus.validation.send` | Full hash of the validated ledger (shared with peer) | -| `full_validation` | boolean | `consensus.validation.send` | Whether this is a full validation | -| `validation_sign_time` | int64 | `consensus.validation.send` | Validation signing time | -| `mode_old` | string | `consensus.mode_change` | Operating mode before the transition | -| `mode_new` | string | `consensus.mode_change` | Operating mode after the transition | +| Attribute | Type | Set On | Description | +| --------------------------- | ------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | +| `consensus_ledger_id` | string | `consensus.round` | Previous-ledger id anchoring the round | +| `ledger_seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept.apply`, `consensus.validation.send` | Ledger sequence number | +| `consensus_mode` | string | `consensus.round`, `consensus.ledger_close` | Node mode: `"Proposing"`, `"Observing"`, `"Wrong"`, etc. | +| `consensus_round_id` | int64 | `consensus.round` | Round identifier | +| `consensus_phase` | string | `consensus.round` | Current phase name (updated on each transition) | +| `trace_strategy` | string | `consensus.round` | Trace-id strategy (`deterministic` / `attribute`) | +| `previous_ledger_seq` | int64 | `consensus.round` | Sequence of the previous ledger | +| `previous_proposers` | int64 | `consensus.round` | Proposer count in the previous round | +| `previous_round_time_ms` | int64 | `consensus.round` | Duration of the previous round | +| `consensus_round` | int64 | `consensus.proposal.send` | Proposal sequence number for the broadcast proposal | +| `is_bow_out` | boolean | `consensus.proposal.send` | Whether the proposal is a bow-out (resigning the round) | +| `tx_count_open` | int64 | `consensus.ledger_close` | Transactions in the open ledger at close | +| `close_time_resolution_ms` | int64 | `consensus.ledger_close` | Close-time rounding granularity | +| `converge_percent` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.check` | Convergence percentage | +| `establish_count` | int64 | `consensus.establish`, `consensus.check` | Establish-phase iteration count | +| `proposers` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.accept` | Number of proposers | +| `disputes_count` | int64 | `consensus.establish`, `consensus.update_positions` | Number of disputed transactions | +| `tx_id` | string | `consensus.update_positions` | Disputed transaction id (per-dispute event) | +| `dispute_our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed tx | +| `dispute_yays` | int64 | `consensus.update_positions` | Yes votes on the disputed tx | +| `dispute_nays` | int64 | `consensus.update_positions` | No votes on the disputed tx | +| `avalanche_threshold` | int64 | `consensus.update_positions` | Escalated weight needed to change our vote | +| `close_time_threshold` | int64 | `consensus.update_positions` | Close-time agreement threshold percentage | +| `agree_count` | int64 | `consensus.check` | Agreeing proposer count | +| `disagree_count` | int64 | `consensus.check` | Disagreeing proposer count | +| `threshold_percent` | int64 | `consensus.check` | Agreement threshold percentage | +| `have_close_time_consensus` | boolean | `consensus.update_positions`, `consensus.check` | Whether the close time reached consensus | +| `proposers_finished` | int64 | `consensus.check` | Proposers that have already validated the next ledger | +| `consensus_stalled` | boolean | `consensus.check` | Whether `checkConsensus` reported a stall | +| `consensus_result` | string | `consensus.check` | Check outcome | +| `quorum` | int64 | `consensus.accept` | Quorum required | +| `round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds | +| `consensus_state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` | +| `close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) | +| `close_time_correct` | boolean | `consensus.accept.apply` | Whether validators agreed on close time | +| `close_resolution_ms` | int64 | `consensus.accept.apply` | Close-time rounding granularity in milliseconds | +| `proposing` | boolean | `consensus.accept.apply`, `consensus.validation.send` | Whether this node was a proposer | +| `parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger close time | +| `close_time_self` | int64 | `consensus.accept.apply` | This node's close-time vote | +| `close_time_vote_bins` | string | `consensus.accept.apply` | Distribution of close-time votes | +| `resolution_direction` | string | `consensus.accept.apply` | Whether close resolution increased/decreased/unchanged | +| `tx_count` | int64 | `consensus.accept.apply` | Transactions in the accepted set | +| `ledger_hash` | string | `consensus.validation.send` | Full hash of the validated ledger (shared with peer) | +| `full_validation` | boolean | `consensus.validation.send` | Whether this is a full validation | +| `validation_sign_time` | int64 | `consensus.validation.send` | Validation signing time | +| `mode_old` | string | `consensus.mode_change` | Operating mode before the transition | +| `mode_new` | string | `consensus.mode_change` | Operating mode after the transition | + +> **`quorum` is on `consensus.accept` only.** Its single set site is +> `RCLConsensus::Adaptor::makeAcceptSpan()` +> (`src/xrpld/app/consensus/RCLConsensus.cpp:516`). `consensus.check` +> (`include/xrpl/consensus/Consensus.h:1899-1926`) never sets it, so +> `{name="consensus.check" && span.quorum>0}` matches nothing. + +> **`consensus.check` carries nine attributes, all set before the early +> returns.** `Consensus::haveConsensus()` sets them at +> `include/xrpl/consensus/Consensus.h:1899-1911` and `consensus_result` at +> `:1926`, deliberately ahead of the `No` / `Expired` branches, so the span is +> fully populated even on rounds that never reach consensus. In set order: +> `agree_count`, `disagree_count`, `converge_percent`, +> `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, +> `consensus_stalled`, `establish_count`, `consensus_result`. +> +> Three of these are shared with sibling spans and were previously scoped too +> narrowly in the table above: `converge_percent` and `establish_count` are set on +> `consensus.check` as well as `consensus.establish` / +> `consensus.update_positions`, and `have_close_time_consensus` is set on both +> `consensus.update_positions` (`Consensus.h:1779`) and `consensus.check` +> (`:1903`). `close_time_threshold` (`:1781`) and `avalanche_threshold` (`:1730`) +> stay `consensus.update_positions`-only. **Tempo query**: `{span.consensus_mode="Proposing"}` to find rounds where the node was proposing. @@ -427,15 +479,24 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=}`. | `tx_count` | int64 | `tx.apply` | Transactions applied to the ledger | | `tx_failed` | int64 | `tx.apply` | Failed transactions in the apply set | | `validations` | int64 | `ledger.validate` | Number of validations received for this ledger | -| `acquire_reason` | string | `ledger.acquire` | Why the ledger fetch was triggered | +| `acquire_reason` | string | `ledger.acquire` | Fetch trigger (`history`/`consensus`/`generic`) | | `timeouts` | int64 | `ledger.acquire` | Number of fetch timeouts | | `peer_count` | int64 | `ledger.acquire` | Peers queried during the fetch | -| `outcome` | string | `ledger.acquire` | Fetch outcome | +| `outcome` | string | `ledger.acquire` | Fetch outcome (`complete`/`failed`/`aborted`) | The apply-step span `tx.apply` (child of `ledger.build`) carries `tx_count`/`tx_failed`; the parent `ledger.build` carries `ledger_seq` and the close-time attributes. `ledger.acquire` (InboundLedger) also sets `ledger_seq`. +`outcome` takes one of **three** values, not two. `complete` and `failed` are both set in +`done()` (`InboundLedger.cpp:530-532`), where `failed` covers both giving up after the +retry limit and hitting unusable ledger data, so a `failed` span can carry `timeouts=0`. `aborted` is set in `~InboundLedger()` when the object is destroyed while +`!isDone()` (`InboundLedger.cpp:242-246`) — the acquisition was **abandoned** before it +finished, rather than having run to its retry limit. The abort path records `timeouts` but +deliberately **not** `peer_count`, because reading the peer count goes through `Overlay`, which +a destructor must not depend on still existing. A query that only groups by +`complete`/`failed` therefore silently loses every abandoned fetch. + **Tempo query**: `{span.ledger_seq=12345}` to find all spans for a specific ledger. #### Peer Attributes @@ -540,7 +601,12 @@ endpoint=http://localhost:4318/v1/metrics prefix=xrpld ``` -Fallback (StatsD): +Fallback (StatsD). `StatsDCollector` is still selected by this value, but the +stack in `docker/telemetry/` no longer receives it: using this path also requires +re-adding the `statsd` receiver to `otel-collector-config.yaml` and uncommenting +port 8125 in `docker-compose.yml`, otherwise the metrics go to a port nothing +listens on. Note also that `StatsDCollector` applies `prefix` to the metric name +while `OTelCollector` does not, so switching transports renames every series. ```ini [insight] @@ -551,25 +617,44 @@ prefix=xrpld ### 2.1 Gauges -| Prometheus Metric | Source File | Description | Typical Range | -| ------------------------------------------- | --------------------- | ----------------------------------------- | ------------------------------- | -| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) | -| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) | -| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative seconds in Disconnected state | Monotonic | -| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative seconds in Connected state | Monotonic | -| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative seconds in Syncing state | Monotonic | -| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative seconds in Tracking state | Monotonic | -| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative seconds in Full state | Monotonic (should dominate) | -| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low | -| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low | -| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low | -| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low | -| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) | -| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 | -| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 | -| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth | -| `overlay_peer_disconnects_charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) | -| `jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) | +| Prometheus Metric | Source File | Description | Typical Range | +| ------------------------------------------- | --------------------- | ------------------------------------------------- | ------------------------------- | +| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) | +| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) | +| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Disconnected state | Monotonic | +| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Connected state | Monotonic | +| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Syncing state | Monotonic | +| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Tracking state | Monotonic | +| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Full state | Monotonic (should dominate) | +| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low | +| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low | +| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low | +| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low | +| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) | +| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 | +| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 | +| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth | +| `jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) | + +> **`state_accounting_*_duration` is microseconds, not seconds.** +> `NetworkOPsImp::collectMetrics()` does +> `duration_cast(...)` and publishes `.count()` +> (`src/xrpld/app/misc/NetworkOPs.cpp:4884-4897`). Divide by `1e6` for seconds. +> The `node-health` "State Duration Rate (All States)" panel already does +> (`/ 1000000` on each `rate(...)`), and +> `docker/telemetry/grafana/dashboards/validate_dashboards.py:44-45` lints the +> family as "cumulative µs". Reading the raw value as seconds overstates time +> in state by a factor of one million. + +> **`overlay_peer_disconnects_charges` was never implemented: NOT IMPLEMENTED.** +> No instrument of that name exists anywhere in `src/`, `include/` or `docker/`. +> The resource-charge disconnect count is exported from the OTel +> `MetricsRegistry` instead, as +> `server_info{metric="peer_disconnects_resources"}` — see +> [§Server Info](#server-info-via-otel-metricsregistry). Use that selector; +> the previously documented `overlay_peer_disconnects_charges` matches nothing. +> `06-implementation-phases.md` still names the old metric in its Phase 6/7 +> task text and panel table. **Grafana dashboard**: _Node Health_ (`node-health`) @@ -686,7 +771,7 @@ types where this bites are the ones with a low concurrency limit > **Sampling caveat.** These are sampled, not integrated. The values are read > when the SDK's periodic reader invokes the observable callbacks, which run the > collector hooks; the export interval is 1000 ms -> (`export_interval_millis` in `src/libxrpl/telemetry/Telemetry.cpp:441`) and +> (`export_interval_millis` in `src/libxrpl/telemetry/Telemetry.cpp:476`) and > hook invocation is debounced to at most once per 500 ms. A spike shorter than > the interval can be missed entirely, so read these as pressure indicators > rather than as exact peak depths. @@ -775,13 +860,16 @@ for how the tier attributes are set and reach metrics. 1. Open Grafana at **http://localhost:3000** 2. Navigate to **Dashboards → xrpld** folder -3. All 15 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/` +3. All 16 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/` + (the workload harness checks that all 16 provision and load; 15 of them also have + metric-data assertions — `log-derived-insights` is Loki-backed, so only its + provisioning is checked) --- ## 4. Tempo Trace Search Guide -> **See also**: [08-appendix.md](./08-appendix.md) §8.2 for span hierarchy visualizations. [05-configuration-reference.md](./05-configuration-reference.md) §5.8.5 for TraceQL query examples. +> **See also**: [08-appendix.md](./08-appendix.md) §8.2 for span hierarchy visualizations. [05-configuration-reference.md](./05-configuration-reference.md) §5.8.4 for TraceQL query examples. ### Finding Traces by Type @@ -813,16 +901,16 @@ A consensus round groups its lifecycle spans under a single root (`consensus.round`); the build/ledger spans run as their own trees: ``` -consensus.round (root — one per round) - ├── consensus.phase.open (open phase) - ├── consensus.proposal.send (broadcast proposal) - ├── consensus.ledger_close (close event) - ├── consensus.establish (establish phase) - ├── consensus.update_positions (position updates) - ├── consensus.check (threshold check) - ├── consensus.accept (accept result) - │ └── consensus.accept.apply (apply, jtACCEPT thread) - └── consensus.validation.send (send validation, follows-from link) +consensus.round (root — one per round) + ├── consensus.phase.open (open phase) + ├── consensus.proposal.send (broadcast proposal) + ├── consensus.ledger_close (close event) + ├── consensus.establish (establish phase) + │ ├── consensus.update_positions (position updates) + │ └── consensus.check (threshold check) + ├── consensus.accept (accept result) + │ └── consensus.accept.apply (apply, jtACCEPT thread) + └── consensus.validation.send (send validation, follows-from link) ledger.build (build new ledger) └── tx.apply (apply transaction set) @@ -834,7 +922,7 @@ ledger.store (persist to DB) ## 5. Prometheus Query Examples -> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8.7 for correlating Prometheus system metrics with trace-derived metrics. +> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8.6 for correlating Prometheus system metrics with trace-derived metrics. ### Span-Derived Metrics @@ -881,7 +969,7 @@ state_accounting_full_duration > **Plan details**: [06-implementation-phases.md §6.8.1](./06-implementation-phases.md) — motivation, architecture, Mermaid diagrams > **Task breakdown**: [Phase8_taskList.md](./Phase8_taskList.md) — per-task implementation details -Phase 8 injects OTel trace context into xrpld's `Logs::format()` output, enabling log-trace correlation. When a log line is emitted within an active OTel span, the trace and span identifiers are automatically appended after the severity field: +Phase 8 injects OTel trace context into xrpld's `Logs::format()` output, enabling log-trace correlation. When a log line is emitted within an active, sampled OTel span, the trace and span identifiers are automatically appended after the severity field: ### Log Format @@ -897,7 +985,7 @@ Example: - **`trace_id=`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo. - **`span_id=`** — 16-character lowercase hex span identifier. Identifies the specific span within the trace. -- **Only present** when the log is emitted within an active OTel span. Log lines outside of traced code paths have no trace context fields. +- **Only present** when the log is emitted within an active OTel span whose context is sampled. Log lines outside of traced code paths, and lines inside a span the sampler dropped, have no trace context fields. A dropped span still carries its parent's identifiers, so emitting them would point at a trace that was never exported. ### Implementation @@ -933,18 +1021,26 @@ Grafana Loki (v3.7.6) serves as the log storage backend. It receives log entries ### LogQL Query Examples +The stream selector is `{service_name="xrpld"}`, **not** `{job="xrpld"}`. Loki's +OTLP ingestion promotes only a small set of resource attributes to stream labels +(`service_name`, `service_instance_id`, `deployment_environment`); everything else +— including the `job` attribute the collector sets — lands in structured +metadata and must be filtered with `|` after the selector. A `{job="xrpld"}` +selector returns zero rows and no error. All shipped queries and the +`log-derived-insights` dashboard use the `service_name` form. + ```logql # Find all logs for a specific trace -{job="xrpld"} |= "trace_id=abc123def456789012345678abcdef01" +{service_name="xrpld"} |= "trace_id=abc123def456789012345678abcdef01" # Error logs with trace context -{job="xrpld"} |= "ERR" |= "trace_id=" +{service_name="xrpld"} |= "ERR" |= "trace_id=" # Logs from a specific partition with trace context -{job="xrpld"} |= "LedgerMaster" | regexp `trace_id=(?P[a-f0-9]+)` | trace_id != "" +{service_name="xrpld"} | partition = `LedgerMaster` | trace_id != "" # Count traced log lines over time -count_over_time({job="xrpld"} |= "trace_id=" [5m]) +count_over_time({service_name="xrpld"} |= "trace_id=" [5m]) ``` --- @@ -961,10 +1057,17 @@ async callbacks for new categories. > **Authoritative metric names live in [§ Phase 9: OTel SDK-Exported Metrics](#phase-9-otel-sdk-exported-metrics-metricsregistry) below.** > Most internal metrics are emitted as **labeled** gauges — one instrument carrying many logical -> values via a `metric` label (e.g. `cache_metrics{metric="sle_hit_rate"}`, +> values via a `metric` label (e.g. `cache_metrics{metric="SLE_hit_rate"}`, > `txq_metrics{metric="txq_count"}`, `load_factor_metrics{metric="load_factor"}`, > `nodestore_state{metric="node_reads_total"}`) — not the flat per-name form. Query the -> labeled names; the flat names (`cache_sle_hit_rate`, `txq_count`, …) are **not** emitted. +> labeled names; the flat names (`cache_SLE_hit_rate`, `txq_count`, …) are **not** emitted. +> +> **Label values are case-sensitive and three cache values are not lowercase.** +> The `metric` label carries the string literal passed to `Observe()`, verbatim: +> `SLE_hit_rate`, `AL_hit_rate` and `AL_size` are upper-case +> (`src/xrpld/telemetry/MetricsRegistry.cpp:666`, `:682`, `:708`), while +> `ledger_hit_rate` genuinely is lowercase (`:675`). A selector written as +> `cache_metrics{metric="sle_hit_rate"}` matches nothing. #### Server Info (via OTel MetricsRegistry) @@ -1006,16 +1109,33 @@ async callbacks for new categories. | Prometheus Metric | Type | Labels | Description | | --------------------------------- | ----- | -------- | ------------------------- | -| `cache_metrics{metric="al_size"}` | Gauge | `metric` | AcceptedLedger cache size | +| `cache_metrics{metric="AL_size"}` | Gauge | `metric` | AcceptedLedger cache size | #### Extended NodeStore Metrics (additions to existing nodestore_state) -| Prometheus Metric | Type | Labels | Description | -| -------------------------------------------------- | ----- | -------- | ----------------------------------- | -| `nodestore_state{metric="node_reads_duration_us"}` | Gauge | `metric` | Cumulative read time (microseconds) | -| `nodestore_state{metric="read_request_bundle"}` | Gauge | `metric` | Read request bundle count | -| `nodestore_state{metric="read_threads_running"}` | Gauge | `metric` | Active read threads | -| `nodestore_state{metric="read_threads_total"}` | Gauge | `metric` | Total read threads configured | +| Prometheus Metric | Type | Labels | Description | +| --------------------------------------------------- | ----- | -------- | ------------------------------------ | +| `nodestore_state{metric="node_reads_duration_us"}` | Gauge | `metric` | Cumulative read time (microseconds) | +| `nodestore_state{metric="node_writes_duration_us"}` | Gauge | `metric` | Cumulative write time (microseconds) | +| `nodestore_state{metric="read_request_bundle"}` | Gauge | `metric` | Read request bundle count | +| `nodestore_state{metric="read_threads_running"}` | Gauge | `metric` | Active read threads | +| `nodestore_state{metric="read_threads_total"}` | Gauge | `metric` | Total read threads configured | + +> **The cumulative duration pair truncates to whole microseconds.** Both values +> are accumulated in nanoseconds internally and divided on read — +> `getFetchDurationUs()` returns `fetchDurationNs_ / 1000` and +> `getStoreDurationUs()` returns `storeDurationNs_ / 1000` +> (`include/xrpl/nodestore/Database.h:232-254`). The exported unit is +> microseconds and every doc, metric and dashboard agrees on that — this is +> **not** a unit mismatch. The consequence is only at the low end: a handful of +> sub-microsecond reads on a warm store can leave the gauge reading `0` until +> their nanosecond total passes 1000. Read a flat `0` on a low-traffic node as +> "not yet a microsecond of I/O", not as "no I/O". +> +> `node_writes_duration_us` is covered by +> `validate_dashboards.py`'s `NODESTORE_CUMULATIVE` tuple, so the raw-counter +> lint would catch a misuse, but it has **no dashboard panel** yet — an open +> follow-up, unlike its `node_reads_duration_us` sibling. #### Job Queue and GetObject Additions @@ -1036,12 +1156,17 @@ repeated here: write-serialized stall from a cold-read stall. See [Sync Diagnosis Signals](#sync-diagnosis-signals-observable-gauge--nodestore_state). -### New Grafana Dashboards (Phase 9) +### New Grafana Dashboards for the Phase 9 Gap-Fill Metrics + +These two boards were created specifically to surface the gap-fill metrics above. +For the full Phase-9 dashboard delivery record, including the boards added to the +Phase-7 parity set, see +[New Grafana Dashboards (Phase 9)](#new-grafana-dashboards-phase-9). | Dashboard | UID | Data Source | Key Panels | | ------------------ | ------------ | ----------- | ----------------------------------------------------------------- | | Fee Market & TxQ | `fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown, escalation | -| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times, queue depth | +| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times, overflow rate | --- @@ -1076,17 +1201,37 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 > below as **families currently emitting** (idle nodes under-report — workload-gated metrics such as > per-RPC/error counters appear only once exercised, which is Phase 10's purpose). -| Category | Expected Count | Validation Method | Config File | -| ------------------------------ | ------------------------- | -------------------------------- | ----------------------- | -| Trace spans | ~37 (required + optional) | Tempo API query | `expected_spans.json` | -| Span attributes | per-span assertion | Per-span attribute assertion | `expected_spans.json` | -| Legacy beast::insight families | ~270 (≈224 traffic) | Prometheus `__name__` query | `expected_metrics.json` | -| Native MetricsRegistry | 35 instruments | Prometheus query | `expected_metrics.json` | -| Call-site `XRPL_METRIC_*` | 7 instruments | Prometheus query | `expected_metrics.json` | -| Per-job-type gauges | 105 (35 types × 3) | Prometheus `__name__` query | `expected_metrics.json` | -| SpanMetrics RED | 4 per span | Prometheus query | `expected_metrics.json` | -| Grafana dashboards | 15 | Dashboard API "no data" check | `expected_metrics.json` | -| Log-trace links | Present | Loki query + Tempo reverse check | — | +| Category | Expected Count | Validation Method | Config File | +| ------------------------------ | ------------------- | -------------------------------- | ----------------------- | +| Trace spans | 40 of 41 emitted | Tempo API query | `expected_spans.json` | +| Span attributes | 67 required | Per-span attribute assertion | `expected_spans.json` | +| Legacy beast::insight families | ~270 (≈224 traffic) | Prometheus `__name__` query | `expected_metrics.json` | +| Native MetricsRegistry | 35 instruments | Prometheus query | `expected_metrics.json` | +| Call-site `XRPL_METRIC_*` | 7 instruments | Prometheus query | `expected_metrics.json` | +| Per-job-type gauges | 105 (35 types × 3) | Prometheus `__name__` query | `expected_metrics.json` | +| SpanMetrics RED | 4 per span | Prometheus query | `expected_metrics.json` | +| Grafana dashboards | all 15 on disk | Dashboard API load + panel count | `expected_metrics.json` | +| Log-trace links | Present | Loki query + Tempo reverse check | — | + +> **These are the harness's numbers, not the code's, and two of them differ.** +> `docker/telemetry/workload/expected_spans.json` carries 40 span entries against +> the **41** families the code emits ([§1.1](#11-complete-span-inventory-41-spans)) — +> `rpc.ws_upgrade` has no entry — and 67 distinct required attributes (the +> manifest's own `total_unique_attributes: 58` field is stale). +> `expected_metrics.json` lists all **15** dashboard uids in +> `docker/telemetry/grafana/dashboards/`, so dashboard coverage does not differ; +> `log-derived-insights` is listed for the provisioning check only, and its panel +> data is asserted nowhere. The 35 native instruments match +> the tables in +> [§Phase 9: OTel SDK-Exported Metrics](#phase-9-otel-sdk-exported-metrics-metricsregistry) +> and the Phase 7+ section exactly, counting each labeled gauge family +> (`nodestore_state`, `cache_metrics`, …) once. +> +> Note that `ledgers_closed_total` appears in **both** instrument rows: it is +> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:386-387`, whose +> `incrementLedgersClosed()` has no callers) and separately incremented at its +> call site via `XRPL_METRIC_COUNTER_INC` (`RCLConsensus.cpp:749`). The distinct +> name count across the two rows is therefore 41, not 42. The two added rows are the families that do not originate as `MetricsRegistry` members. **Call-site** instruments are declared by the `XRPL_METRIC_*` macros @@ -1208,7 +1353,7 @@ via OTLP/HTTP to the OTel Collector and scraped by Prometheus. > **On NuDB, `write_load` and `nudb_writers_in_flight` are the same number.** > Both read the same atomic. `NuDBBackend::getWriteLoad()` returns > `concurrentWriters.load()` -> (`src/libxrpl/nodestore/backend/NuDBFactory.cpp:355-361`), and +> (`src/libxrpl/nodestore/backend/NuDBFactory.cpp:375-381`), and > `WriteStats::concurrentWriters` is that same counter. So the two series track > each other exactly, sampled microseconds apart in one callback. Do not read > their agreement as two signals confirming each other — it is one signal twice. @@ -1225,7 +1370,7 @@ Further label values on the same instrument, added to separate the two bottlenecks that both present as the `ledgerData` job lane pinned at its concurrency cap. Observed in `MetricsRegistry::observeNodeStoreTotals()`, `observeWritePathDetail()`, and `observeAcquireStats()` -(`src/xrpld/telemetry/MetricsRegistry.cpp:805-877`). +(`src/xrpld/telemetry/MetricsRegistry.cpp:871-942`). | Prometheus Metric | Type | Labels | Description | | ---------------------------------------------------- | ----- | -------- | ------------------------------------------------------- | @@ -1294,9 +1439,9 @@ data as uninformative unless the build is known to include the fix. | Prometheus Metric | Type | Labels | Description | | --------------------------------------------- | ----- | -------- | ----------------------------- | -| `cache_metrics{metric="sle_hit_rate"}` | Gauge | `metric` | SLE cache hit rate (0.0-1.0) | +| `cache_metrics{metric="SLE_hit_rate"}` | Gauge | `metric` | SLE cache hit rate (0.0-1.0) | | `cache_metrics{metric="ledger_hit_rate"}` | Gauge | `metric` | Ledger cache hit rate | -| `cache_metrics{metric="al_hit_rate"}` | Gauge | `metric` | AcceptedLedger cache hit rate | +| `cache_metrics{metric="AL_hit_rate"}` | Gauge | `metric` | AcceptedLedger cache hit rate | | `cache_metrics{metric="treenode_cache_size"}` | Gauge | `metric` | SHAMap TreeNode cache entries | | `cache_metrics{metric="treenode_track_size"}` | Gauge | `metric` | Tracked tree nodes | | `cache_metrics{metric="fullbelow_size"}` | Gauge | `metric` | FullBelow cache entries | @@ -1314,6 +1459,73 @@ data as uninformative unless the build is known to include the fix. | `txq_metrics{metric="txq_med_fee_level"}` | Gauge | `metric` | Median fee level in queue | | `txq_metrics{metric="txq_open_ledger_fee_level"}` | Gauge | `metric` | Open ledger fee escalation level | +#### TxQ Admission and Ledger Mismatch (Synchronous Counters) + +Three monotonic counters created alongside the Phase 7+ parity counters +(`src/xrpld/telemetry/MetricsRegistry.cpp:394-399`). The gauges above answer +"how deep is the queue"; these answer "what did the queue refuse, and did the +ledger we built match the one the network validated". + +| Prometheus Metric | Type | Labels | Description | Increment Site | +| ------------------------------- | ------- | ----------------- | -------------------------------------------------- | --------------------- | +| `txq_dropped_total` | Counter | `reason=""` | Transactions refused admission to the queue | TxQ.cpp:1302,1347 | +| `txq_expired_total` | Counter | (none) | Transactions abandoned out of the queue on expiry | TxQ.cpp:1428 | +| `ledger_history_mismatch_total` | Counter | `reason=""` | Built-vs-validated ledger hash mismatches, by kind | LedgerHistory.cpp:332 | + +Label domains, as emitted: + +| Label | Values | +| --------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `txq_dropped_total{reason}` | `queue_full` | +| `ledger_history_mismatch_total{reason}` | `prior_ledger`, `close_time`, `consensus_txset`, `different_txset`, `same_txset_diff_result`, `unknown` | + +**Grafana dashboards**: _Fee Market & TxQ_ (`fee-market`) — "Queue Admission +Rejections (Dropped)", "Queue Abandonment Rate (Expired)"; _Consensus Health_ +(`consensus-health`) — "Ledger History Mismatch Rate by Reason"; _Node Health_ +(`node-health`) — "Ledger History Mismatches". + +> **Known issue — `ledger_history_mismatch_total` has two producers, so a bare +> `sum()` double-counts.** `LedgerHistory::handleMismatch()` increments **both** +> a `beast::insight` counter registered as `ledger.history` / `mismatch` +> (`src/xrpld/app/ledger/LedgerHistory.cpp:323`, created at `:41`) **and** the +> OTel counter above (`:331-332`). The insight counter carries **no** `reason` +> label, and the Prometheus exporter appends `_total` to both, so the two land in +> one metric family: per-node series carrying a `reason` label, plus per-node +> series with `reason` absent that already total all of them. The dual-producer +> mechanism is verifiable from the code above; the exact series count in any given +> stack depends on how many nodes report and how many distinct reasons they have +> hit, so do not treat a fixed number as an invariant. +> +> Consequence: `sum(rate(ledger_history_mismatch_total[5m]))` counts every +> mismatch twice. Always group or filter by `reason`: +> `sum by (reason) (rate(ledger_history_mismatch_total{reason!=""}[5m]))` for the +> per-reason breakdown, or `reason=""` for the untyped total alone. This is a +> **code** defect, not a documentation one — the fix is to retire one producer; +> until then the shipped panels avoid the trap (`consensus-health` groups +> `by (reason)`, `node-health` plots the series unaggregated), and any new panel +> or alert must do the same. + +#### Reduce-Relay Efficiency (Observable Gauge — `reduce_relay_metrics`) + +Transaction reduce-relay effectiveness, read from `Overlay::txMetrics()` each +collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1370-1402`). A high +`suppressed_peers` : `selected_peers` ratio proves the feature is saving +bandwidth; a high `not_enabled_peers` means stale peers are forcing full relay. + +| Prometheus Metric | Type | Labels | Description | +| -------------------------------------------------- | ----- | -------- | ------------------------------------------------------- | +| `reduce_relay_metrics{metric="selected_peers"}` | Gauge | `metric` | Peers selected to receive a relayed transaction | +| `reduce_relay_metrics{metric="suppressed_peers"}` | Gauge | `metric` | Peer sends suppressed by reduce-relay | +| `reduce_relay_metrics{metric="not_enabled_peers"}` | Gauge | `metric` | Peers without reduce-relay support, so relayed in full | +| `reduce_relay_metrics{metric="missing_tx_freq"}` | Gauge | `metric` | Frequency of transactions this node had to request back | + +Each source field is a decimal **string** in the `txMetrics()` JSON, parsed with +`std::stoll`; a field that is absent or unparseable is skipped rather than +reported as zero, so absent is not zero here either. + +**Grafana dashboard**: _Peer Network_ (`peer-network`) — "Reduce-Relay Peer +Selection", "Reduce-Relay Missing-Tx Frequency". + #### Per-RPC Method Metrics (Synchronous Counters/Histogram) | Prometheus Metric | Type | Labels | Description | @@ -1456,17 +1668,43 @@ spelling would silently drop the override. #### Counted Object Instances (Observable Gauge — `object_count`) -| Prometheus Metric | Type | Labels | Description | -| -------------------------------------- | ----- | --------------- | ------------------------------ | -| `object_count{type="transaction"}` | Gauge | `type=""` | Live Transaction objects | -| `object_count{type="ledger"}` | Gauge | `type=""` | Live Ledger objects | -| `object_count{type="nodeobject"}` | Gauge | `type=""` | Live NodeObject instances | -| `object_count{type="sttx"}` | Gauge | `type=""` | Serialized transaction objects | -| `object_count{type="stledgerentry"}` | Gauge | `type=""` | Serialized ledger entries | -| `object_count{type="inboundledger"}` | Gauge | `type=""` | Ledgers being fetched | -| `object_count{type="pathfinder"}` | Gauge | `type=""` | Active pathfinding operations | -| `object_count{type="pathrequest"}` | Gauge | `type=""` | Active path requests | -| `object_count{type="hashrouterentry"}` | Gauge | `type=""` | Hash router entries | +**The `type` label value is the demangled, fully-qualified C++ type name.** It is +not a lowercase word and not a friendly alias. The value is +`beast::typeName()` (`include/xrpl/basics/CountedObject.h:115`), which +demangles `typeid(T).name()` with `abi::__cxa_demangle` +(`include/xrpl/beast/type_name.h:16-45`) and applies no stripping; the observer +copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:781-787`). +Values therefore keep their `xrpl::` namespace, nested `::`, and template +arguments. + +| Prometheus Metric | Type | Labels | Description | +| ---------------------------------------------- | ----- | --------------- | ------------------------------ | +| `object_count{type="xrpl::Transaction"}` | Gauge | `type=""` | Live Transaction objects | +| `object_count{type="xrpl::Ledger"}` | Gauge | `type=""` | Live Ledger objects | +| `object_count{type="xrpl::NodeObject"}` | Gauge | `type=""` | Live NodeObject instances | +| `object_count{type="xrpl::STTx"}` | Gauge | `type=""` | Serialized transaction objects | +| `object_count{type="xrpl::STLedgerEntry"}` | Gauge | `type=""` | Serialized ledger entries | +| `object_count{type="xrpl::InboundLedger"}` | Gauge | `type=""` | Ledgers being fetched | +| `object_count{type="xrpl::Pathfinder"}` | Gauge | `type=""` | Active pathfinding operations | +| `object_count{type="xrpl::PathRequest"}` | Gauge | `type=""` | Active path requests | +| `object_count{type="xrpl::HashRouter::Entry"}` | Gauge | `type=""` | Hash router entries | + +The list above is the subset most often queried, not the whole label domain. The +series set is whatever `CountedObject` subclasses have been instantiated, so +it also includes `xrpl::SHAMapItem`, `xrpl::SHAMapInnerNode`, +`xrpl::AcceptedLedger`, `xrpl::Job`, template instantiations such as +`xrpl::STBitString<256>` and `xrpl::STInteger`, and a few types +outside the `xrpl` namespace such as `CachedView::hit`. Enumerate it rather than +guess: + +```promql +# Every type currently reporting on one node +count by (type) (object_count{service_instance_id=~"$node"}) +``` + +Grafana's `$type` template variable on _Node Health_ is populated the same way +(`label_values(object_count, type)`), which is why that dashboard needs no +hardcoded list. #### Load Factor Breakdown (Observable Gauge — `load_factor_metrics`) @@ -1559,30 +1797,51 @@ These metrics fill gaps identified by comparing xrpld's internal observability w | -------------------------------------------------- | ------ | -------- | --------------------------------------- | | `validation_agreement{metric="agreement_pct_1h"}` | Double | `metric` | Rolling 1h agreement percentage (0-100) | | `validation_agreement{metric="agreement_pct_24h"}` | Double | `metric` | Rolling 24h agreement percentage | -| `validation_agreement{metric="agreements_1h"}` | Int64 | `metric` | Agreed validations in 1h window | -| `validation_agreement{metric="missed_1h"}` | Int64 | `metric` | Missed validations in 1h window | -| `validation_agreement{metric="agreements_24h"}` | Int64 | `metric` | Agreed validations in 24h window | -| `validation_agreement{metric="missed_24h"}` | Int64 | `metric` | Missed validations in 24h window | +| `validation_agreement{metric="agreement_pct_7d"}` | Double | `metric` | Rolling 7-day agreement percentage | +| `validation_agreement{metric="agreements_1h"}` | Double | `metric` | Agreed validations in 1h window | +| `validation_agreement{metric="missed_1h"}` | Double | `metric` | Missed validations in 1h window | +| `validation_agreement{metric="agreements_24h"}` | Double | `metric` | Agreed validations in 24h window | +| `validation_agreement{metric="missed_24h"}` | Double | `metric` | Missed validations in 24h window | +| `validation_agreement{metric="agreements_7d"}` | Double | `metric` | Agreed validations in the 7-day window | +| `validation_agreement{metric="missed_7d"}` | Double | `metric` | Missed validations in the 7-day window | Data source: `ValidationTracker` class with 8s grace period and 5m late repair window. +> **Every value on this instrument is a double.** The family is one +> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1593`), +> so the integral counts are cast to `double` before `Observe()` — there is no +> Int64 sub-series to filter on. The same holds for `validator_health`, +> `peer_quality` and `state_tracking` below; an earlier revision of these four +> tables split the Type column between Int64 and Double, which the code does not +> do. +> +> The 7-day window is `ValidationTracker::kWindow7d` = 168 hours +> (`src/xrpld/telemetry/ValidationTracker.h:311`) and is observed alongside the 1h +> and 24h windows at `MetricsRegistry.cpp:1623-1626`. Panels exist on _Validator +> Health_ (`validator-health`): "Agreement % (7d)" and "Agreements vs Missed +> (7d)". + #### Validator Health (Observable Gauge — `validator_health`) | Prometheus Metric | Type | Labels | Description | | ---------------------------------------------- | ------ | -------- | ------------------------------ | -| `validator_health{metric="amendment_blocked"}` | Int64 | `metric` | 1 if amendment-blocked, else 0 | -| `validator_health{metric="unl_blocked"}` | Int64 | `metric` | 1 if UNL-blocked, else 0 | +| `validator_health{metric="amendment_blocked"}` | Double | `metric` | 1 if amendment-blocked, else 0 | +| `validator_health{metric="unl_blocked"}` | Double | `metric` | 1 if UNL-blocked, else 0 | | `validator_health{metric="unl_expiry_days"}` | Double | `metric` | Days until UNL list expires | -| `validator_health{metric="validation_quorum"}` | Int64 | `metric` | Validation quorum threshold | +| `validator_health{metric="validation_quorum"}` | Double | `metric` | Validation quorum threshold | + +Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`. #### Peer Quality (Observable Gauge — `peer_quality`) | Prometheus Metric | Type | Labels | Description | | ------------------------------------------------- | ------ | -------- | ------------------------------------ | | `peer_quality{metric="peer_latency_p90_ms"}` | Double | `metric` | P90 peer latency in milliseconds | -| `peer_quality{metric="peers_insane_count"}` | Int64 | `metric` | Peers with diverged tracking status | +| `peer_quality{metric="peers_insane_count"}` | Double | `metric` | Peers with diverged tracking status | | `peer_quality{metric="peers_higher_version_pct"}` | Double | `metric` | % of peers on newer xrpld version | -| `peer_quality{metric="upgrade_recommended"}` | Int64 | `metric` | 1 if >60% of peers are newer version | +| `peer_quality{metric="upgrade_recommended"}` | Double | `metric` | 1 if >60% of peers are newer version | + +Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`. #### Ledger Economy (Observable Gauge — `ledger_economy`) @@ -1598,10 +1857,12 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w | Prometheus Metric | Type | Labels | Description | | -------------------------------------------------------- | ------ | -------- | -------------------------------------- | -| `state_tracking{metric="state_value"}` | Int64 | `metric` | Numeric state 0-6 (see encoding below) | +| `state_tracking{metric="state_value"}` | Double | `metric` | Numeric state 0-6 (see encoding below) | | `state_tracking{metric="time_in_current_state_seconds"}` | Double | `metric` | Duration in current state | -State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). +Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1483`. + +State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1500-1515`. **The range is 0-6, not 0-7** — there is no seventh state. #### Storage Detail (Observable Gauge — `storage_detail`) @@ -1610,11 +1871,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) | > **`stored_object_bytes` is not a file size.** It observes `getStoreSize()` -> (`src/xrpld/telemetry/MetricsRegistry.cpp:1511`), which sums the object payloads +> (`src/xrpld/telemetry/MetricsRegistry.cpp:1574`), which sums the object payloads > this process has written. It therefore excludes NuDB's keys, bucket padding and > log, and it resets when the process restarts while the files on disk do not. > `node_written_bytes` on the `nodestore_state` gauge calls the same accessor -> (`MetricsRegistry.cpp:836`), so the two series are equal by construction and any +> (`MetricsRegistry.cpp:877`), so the two series are equal by construction and any > write-amplification ratio built from the pair is a constant 1.0. To size the store > on disk, stat the backend's files; no metric reports it today. > @@ -1632,6 +1893,24 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `validations_checked_total` | Counter | Network validations observed | LedgerMaster.cpp | | `state_changes_total` | Counter | Operating mode transitions | NetworkOPs.cpp | +> **Known issue — `ledgers_closed_total` has a dead second producer.** The +> instrument is created twice. `MetricsRegistry::registerCounters()` eagerly +> creates it as the member `ledgersClosedCounter_` +> (`src/xrpld/telemetry/MetricsRegistry.cpp:386-387`), and its only mutator, +> `MetricsRegistry::incrementLedgersClosed()` +> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1703`), has +> **zero callers** — the header says so itself at `MetricsRegistry.h:584-588`. +> The value operators actually see comes from the single live increment, +> the `XRPL_METRIC_COUNTER_INC` call site in +> `RCLConsensus::Adaptor::doAccept()` (`src/xrpld/app/consensus/RCLConsensus.cpp:749`). +> +> No metric is wrong and nothing double-counts: the dead member never adds to the +> series. The cost is a redundant eagerly-created instrument plus a misleading API +> that looks like the increment path. **Code follow-up**: delete +> `incrementLedgersClosed()` and `ledgersClosedCounter_` once the macro path is +> considered proven, per the header note. Tracked here rather than fixed in a doc +> pass — the doc is not reworded to imply the member is used. + Lifetime tallies exported as monotonic **ObservableCounters** (not synchronous counters), observed from an existing cumulative source each collection cycle: @@ -1647,19 +1926,40 @@ counters), observed from an existing cumulative source each collection cycle: > decrease) and additive (`agreements_total + missed_total` = ledgers reconciled). The > repair-aware, windowed view remains on `validation_agreement{metric="…"}`. -#### Span Attribute Enrichments (Phases 2-4) +#### Span Attribute Enrichments (Phases 2-4): REMOVED -| Span Name | New Attribute | Type | Source | -| --------------------------- | ------------------------------------ | ------ | ------------------------ | -| `rpc.command.*` | `xrpl.node.amendment_blocked` | bool | Phase 2 — RPCHandler.cpp | -| `rpc.command.*` | `xrpl.node.server_state` | string | Phase 2 — RPCHandler.cpp | -| `tx.receive` | `xrpl.peer.version` | string | Phase 3 — PeerImp.cpp | -| `consensus.validation.send` | `xrpl.validation.ledger_hash` | string | Phase 4 — RCLConsensus | -| `consensus.validation.send` | `xrpl.validation.full` | bool | Phase 4 — RCLConsensus | -| `peer.validation.receive` | `xrpl.peer.validation.ledger_hash` | string | Phase 4 — PeerImp.cpp | -| `peer.validation.receive` | `xrpl.peer.validation.full` | bool | Phase 4 — PeerImp.cpp | -| `consensus.accept` | `xrpl.consensus.validation_quorum` | int64 | Phase 4 — RCLConsensus | -| `consensus.accept` | `xrpl.consensus.proposers_validated` | int64 | Phase 4 — RCLConsensus | +This section used to list nine dotted `xrpl.node.*` / `xrpl.peer.*` / +`xrpl.validation.*` / `xrpl.consensus.*` **span** attributes. **None of them +exists.** A grep for `xrpl.node.`, `xrpl.peer.`, `xrpl.validation.` and +`xrpl.consensus.` across non-test `src/` and `include/` returns nothing, and the +table also contradicted this document's own rule in +[§1.2](#12-complete-attribute-inventory-bareunderscore-keys): dotted keys are +OTel **resource** attributes, never span attributes. + +The dotted form was dropped by the 2026-05-13 naming redesign, in three commits: + +| Commit | Scope | +| ------------ | ------------------------------------------------------------------------------------------------------------- | +| `e339ba1f6b` | tx / txq — dropped the `xrpl..` prefix (phase-3) | +| `46d1012ad4` | consensus — dropped the `xrpl.consensus.` prefix (phase-4) | +| `9e27120a15` | ledger / peer — simplified the keys, updated dashboards (phase-6) | + +What the code emits today, and where it is documented: + +| Old dotted key (never emitted) | Live equivalent | +| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) | +| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` | +| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans | +| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only | +| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1233`) | +| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1031`) | +| `xrpl.consensus.proposers_validated` | **never implemented** in any form | + +The identical nine-row list was deleted from +`docker/telemetry/workload/expected_spans.json` by commit `cb9fce6890` for the +same reason. Anything still asserting these keys — a dashboard filter, a TraceQL +query, an alert — matches nothing and should be pointed at the live keys above. ### New Grafana Dashboards (Phase 9) @@ -1673,10 +1973,9 @@ counters), observed from an existing cumulative source each collection cycle: ### Updated Grafana Dashboards (Phase 9) -| Dashboard | UID | New Panels Added | -| -------------------- | -------------------------- | -------------------------------------------------------------------- | -| Node Health (StatsD) | `xrpld-statsd-node-health` | NodeStore I/O, cache hit rates, object instance counts | -| System Node Health | `node-health` | Ledger economy row: base fee, reserves, ledger age, transaction rate | +| Dashboard | UID | New Panels Added | +| ----------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Node Health | `node-health` | NodeStore I/O row, cache hit rates, object instance counts; Ledger Economy row: base fee, reserves, ledger age, transaction rate | ### New Grafana Dashboards (Phase 11) @@ -1707,20 +2006,24 @@ counters), observed from an existing cumulative source each collection cycle: ## 6. Known Issues -| Issue | Impact | Status | -| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| `warn` and `drop` metrics use non-standard StatsD `\|m` meter type | Metrics silently dropped by OTel StatsD receiver | Phase 6 Task 6.1 — needs `\|m` → `\|c` change in StatsDCollector.cpp | -| `jobq_job_count` may not emit in standalone mode | Missing from Prometheus in some test configs | Requires active job queue activity | -| `rpc_requests` depends on `[insight]` config | Zero series if StatsD not configured | Requires `[insight] server=statsd` in xrpld.cfg | -| Peer tracing enabled by default | `peer.*` spans emit unless `trace_peer=0` | High volume — set `trace_peer=0` to opt out on busy mainnet nodes | -| `handler="other"` mixes several producers | Cannot separate `GetConsL1` from `GetConsL2` | By design — the cardinality bound; see [§Per-Job-Type Metrics](#per-job-type-metrics-synchronous-countershistogram) | -| `overhead_cluster_*` is always zero | 8 dashboard panel references are flatlines by construction; cluster traffic is counted as `unknown` | **NOT IMPLEMENTED** — see [§6.0](#60-mtcluster-is-counted-as-unknown-not-implemented) | -| `squelch_ignored_bytes_in/out` always read zero | Only the `_messages_*` pair carries signal for this category | **NOT IMPLEMENTED** — see [§6.1](#61-squelch_ignored-byte-counts-not-implemented) | -| `total_bytes_in` and `total_bytes_out` use different size bases | In/out byte totals are not directly comparable when compression is on | **NOT IMPLEMENTED** — see [§6.2](#62-inboundoutbound-byte-basis-asymmetry-not-implemented) | -| `overhead` conflates `mtPING` with `mtSTATUS_CHANGE` | Keepalive traffic cannot be isolated from status-change traffic | **NOT IMPLEMENTED** — needs a new category; see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) | -| No metrics for ping RTT distribution, ping timeouts, or `mtENDPOINTS` | Peer keepalive and discovery health are not observable | **NOT IMPLEMENTED** — see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) | -| 11 of 13 peer message families have no spans | `02` §2.3.2 catalogs `peer.message.*`, `peer.connect`, `peer.disconnect` that were never built | **NOT IMPLEMENTED** — see [§6.4](#64-peer-span-coverage-gap-not-implemented) | -| PeerFinder exports 2 of ~17 available slot/cache readings | Slot pressure, connection churn and discovery-cache health are not observable | **NOT IMPLEMENTED** — see [§6.5](#65-peerfinder-slot-and-cache-metrics-not-implemented) | +| Issue | Impact | Status | +| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `warn` and `drop` metrics use non-standard StatsD `\|m` meter type | Metrics silently dropped by OTel StatsD receiver | Phase 6 Task 6.1 — needs `\|m` → `\|c` change in StatsDCollector.cpp | +| `jobq_job_count` may not emit in standalone mode | Missing from Prometheus in some test configs | Requires active job queue activity | +| `rpc_requests` depends on `[insight]` config | Zero series if `[insight]` is absent or unset | Requires `[insight] server=otel` in xrpld.cfg | +| Peer tracing enabled by default | `peer.*` spans emit unless `trace_peer=0` | High volume — set `trace_peer=0` to opt out on busy mainnet nodes | +| `handler="other"` mixes several producers | Cannot separate `GetConsL1` from `GetConsL2` | By design — the cardinality bound; see [§Per-Job-Type Metrics](#per-job-type-metrics-synchronous-countershistogram) | +| `overhead_cluster_*` is always zero | 8 dashboard panel references are flatlines by construction; cluster traffic is counted as `unknown` | **NOT IMPLEMENTED** — see [§6.0](#60-mtcluster-is-counted-as-unknown-not-implemented) | +| `squelch_ignored_bytes_in/out` always read zero | Only the `_messages_*` pair carries signal for this category | **NOT IMPLEMENTED** — see [§6.1](#61-squelch_ignored-byte-counts-not-implemented) | +| `total_bytes_in` and `total_bytes_out` use different size bases | In/out byte totals are not directly comparable when compression is on | **NOT IMPLEMENTED** — see [§6.2](#62-inboundoutbound-byte-basis-asymmetry-not-implemented) | +| `overhead` conflates `mtPING` with `mtSTATUS_CHANGE` | Keepalive traffic cannot be isolated from status-change traffic | **NOT IMPLEMENTED** — needs a new category; see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) | +| No metrics for ping RTT distribution, ping timeouts, or `mtENDPOINTS` | Peer keepalive and discovery health are not observable | **NOT IMPLEMENTED** — see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) | +| 11 of 13 peer message families have no spans | `02` §2.3.2 catalogs `peer.message.*`, `peer.connect`, `peer.disconnect` that were never built | **NOT IMPLEMENTED** — see [§6.4](#64-peer-span-coverage-gap-not-implemented) | +| PeerFinder exports 2 of ~17 available slot/cache readings | Slot pressure, connection churn and discovery-cache health are not observable | **NOT IMPLEMENTED** — see [§6.5](#65-peerfinder-slot-and-cache-metrics-not-implemented) | +| `ledger_history_mismatch_total` has two producers in one family | A bare `sum()` double-counts every mismatch; one series carries no `reason` label | **CODE BUG** — retire one producer; group by `reason` meanwhile. See [§TxQ Admission and Ledger Mismatch](#txq-admission-and-ledger-mismatch-synchronous-counters) | +| `overlay_peer_disconnects_charges` never existed | The documented selector matches nothing; use `server_info{metric="peer_disconnects_resources"}` | **NOT IMPLEMENTED** — see [§2.1](#21-gauges) | +| Nine dotted `xrpl..*` span attributes never shipped | TraceQL filters and harness assertions on the dotted keys match nothing | **NOT IMPLEMENTED** — renamed to bare keys; see [§Span Attribute Enrichments](#span-attribute-enrichments-phases-2-4-removed) | +| `node_writes_duration_us` has no dashboard panel | Cumulative write latency is exported and linted, but never charted | Open follow-up — see [§Extended NodeStore Metrics](#extended-nodestore-metrics-additions-to-existing-nodestore_state) | ### 6.0 `mtCLUSTER` is counted as `unknown`: NOT IMPLEMENTED @@ -1822,9 +2125,9 @@ messages are traced. **Status**: NOT IMPLEMENTED. The span catalog in `02` §2.3.2 is a design inventory, not a statement of what emits; §2.3.2 now marks which entries are -live. Instrumenting the remaining families would change the "~37 spans" count -asserted in [§1.1](#11-complete-span-inventory-37-spans) and in -`docker/telemetry/workload/expected_spans.json`, so it is scoped as its own +live. Instrumenting the remaining families would change the **41 span families** +counted in [§1.1](#11-complete-span-inventory-41-spans) and the **40** catalogued +in `docker/telemetry/workload/expected_spans.json`, so it is scoped as its own change rather than folded into a metric task. ### 6.5 PeerFinder slot and cache metrics: NOT IMPLEMENTED @@ -1838,7 +2141,7 @@ discovery cache has any instrument at all. | Reading | Source | Why it matters | | --------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------- | -| `attempts()`, `attemptsNeeded()` | `Counts.h:68,79` | Outbound connection churn; distinguishes "not trying" from "trying and failing" | +| `attemptsNeeded()`, `attempts()` | `Counts.h:68,79` | Outbound connection churn; distinguishes "not trying" from "trying and failing" | | `outMax()`, `outActive()`, `outboundSlotsFree()` | `Counts.h:88,98,205` | Outbound slot saturation | | `inMax()`, `inboundActive()`, `inboundSlotsFree()` | `Counts.h:165,174,193` | Inbound slot saturation — the two exported gauges give the actives but not the caps, so utilization cannot be computed | | `acceptCount()`, `connectCount()`, `closingCount()` | `Counts.h:138,147,156` | Handshake pipeline depth; `closingCount()` rising is teardown backpressure | @@ -1887,8 +2190,8 @@ The telemetry system is designed with privacy in mind: enabled=1 [insight] -server=statsd -address=127.0.0.1:8125 +server=otel +endpoint=http://localhost:4318/v1/metrics prefix=xrpld ``` @@ -1903,8 +2206,8 @@ batch_size=1024 max_queue_size=4096 [insight] -server=statsd -address=otel-collector:8125 +server=otel +endpoint=http://otel-collector:4318/v1/metrics prefix=xrpld ``` diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index b7289d5004..f2855643a8 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -102,12 +102,16 @@ flowchart TB | **2** | [Design Decisions](./02-design-decisions.md) | SDK selection, exporters, span naming, attributes, context propagation | | **3** | [Implementation Strategy](./03-implementation-strategy.md) | Directory structure, key principles, performance optimization | | **5** | [Configuration Reference](./05-configuration-reference.md) | xrpld config, CMake integration, Collector configurations | -| **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics | +| **6** | [Implementation Phases](./06-implementation-phases.md) | 11-phase timeline, tasks, risks, success metrics | | **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture | | **8** | [Appendix](./08-appendix.md) | Glossary, references, version history | | **9** | [Data Collection Reference](./09-data-collection-reference.md) | Complete inventory of spans, attributes, metrics, and dashboards | | **Sec** | [Securing the OTel Pipeline](./secure-OTel.md) | Threat model and hardening (mTLS, peer trace-context validation) | +> Note there is no document 4: `04-code-samples.md` was removed during the +> rollout, and the numbering was left as-is rather than renumbering every +> cross-reference in the chain. + --- ## 0. Tracing Fundamentals @@ -136,9 +140,9 @@ Key trace points span across transaction submission via RPC, peer-to-peer messag The OpenTelemetry C++ SDK is selected for its CNCF backing, active development, and native performance characteristics. Traces are exported via OTLP/HTTP to an OpenTelemetry Collector, which provides flexible routing and sampling. OTLP/gRPC is planned future work (see design decisions §2.2.2). -Span naming follows a hierarchical `.` convention (e.g., `rpc.submit`, `tx.relay`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs. +Span naming follows a hierarchical `.` convention (e.g., `rpc.command.server_info`, `tx.process`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs. -**Data Collection & Privacy**: Telemetry collects only operational metadata (timing, counts, hashes) — never sensitive content (private keys, balances, amounts, raw payloads). Privacy protection includes account hashing, configurable redaction, sampling, and collector-level filtering. Node operators retain full control over telemetry configuration. +**Data Collection & Privacy**: Telemetry collects only operational metadata (timing, counts, hashes) — never sensitive content (private keys, balances, amounts, raw payloads). Account addresses are hashed **unconditionally** by the SDK helper and hashed again at the collector; there is no redaction config key and therefore no insecure-by-default state. Trace volume is _not_ reduced on the node (head sampling is fixed at 100%); reduction, where wanted, is a collector-side tail-sampling decision. Node operators control which subsystems are traced via the `[telemetry]` per-component toggles. ➡️ **[Read full Design Decisions](./02-design-decisions.md)** @@ -146,9 +150,9 @@ Span naming follows a hierarchical `.` convention (e.g., ` ## 3. Implementation Strategy -The telemetry code is organized under `include/xrpl/telemetry/` for headers and `src/libxrpl/telemetry/` for implementation. Key principles include RAII-based span management via `SpanGuard` (with `discard()` for dropping unwanted spans), a `FilteringSpanProcessor` that intercepts `OnEnd()` to prevent discarded spans from entering the export pipeline, conditional compilation with `XRPL_ENABLE_TELEMETRY`, and minimal runtime overhead through batch processing and efficient sampling. +The telemetry code is organized under `include/xrpl/telemetry/` for headers, `src/libxrpl/telemetry/` for implementation, and `src/xrpld/telemetry/` for the native-metrics module added in Phases 7 and 9. Key principles include RAII-based span management via `SpanGuard` (with `discard()` for dropping unwanted spans), a `FilteringSpanProcessor` that intercepts `OnEnd()` to prevent discarded spans from entering the export pipeline, conditional compilation behind the `XRPL_ENABLE_TELEMETRY` compile definition (set by the CMake `telemetry` option, which defaults to **ON** — build it out with `-Dtelemetry=OFF`), and minimal runtime overhead through batch processing. -Performance optimization strategies include head sampling fixed at 100% (intentionally not configurable, so trace keep/drop decisions stay coherent across nodes), tail-based sampling at the collector for errors and slow traces to reduce volume, batch export to reduce network overhead, and conditional instrumentation that compiles to no-ops when disabled. +Performance optimization strategies include head sampling fixed at 100% (intentionally not configurable, so trace keep/drop decisions stay coherent across nodes), optional tail-based sampling at the collector to reduce stored volume (not enabled in the base stack — the only shipped policy is a 0.5% probabilistic one in the Grafana Cloud overlay), batch export to reduce network overhead, and conditional instrumentation that compiles to no-ops when disabled. ➡️ **[Read full Implementation Strategy](./03-implementation-strategy.md)** @@ -158,9 +162,19 @@ Performance optimization strategies include head sampling fixed at 100% (intenti > **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring -Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, exporter selection, endpoint configuration, and component-level filtering. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration includes a `XRPL_ENABLE_TELEMETRY` option for compile-time control. +Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, TLS/mTLS, batch tuning, and component-level filtering. Exporter selection is _not_ configurable — OTLP/HTTP is the only transport. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration uses the `telemetry` option (default **ON**) for compile-time control. -OpenTelemetry Collector configurations are provided for development and production (with tail-based sampling, Tempo, and Elastic APM). Docker Compose examples enable quick local development environment setup. +Endpoints are spread across **three** keys in two sections, not one "traces and metrics" pair: + +| Signal | Key | Default | Source | +| ---------------------------------------------------- | ------------------------------ | ---------------------------------- | --------------------------- | +| Traces | `[telemetry] endpoint` | `http://localhost:4318/v1/traces` | `TelemetryConfig.cpp:36,61` | +| Native metrics (`XRPL_METRIC_*` / `MetricsRegistry`) | `[telemetry] metrics_endpoint` | `http://localhost:4318/v1/metrics` | `Application.cpp:1670` | +| `beast::insight` metrics (`server=otel`) | `[insight] endpoint` | `http://localhost:4318/v1/metrics` | `CollectorManager.cpp:50` | + +`[telemetry]` itself has exactly **one** `endpoint` key, and it is traces-only. + +The repo ships one collector config (`docker/telemetry/otel-collector-config.yaml`, three pipelines: traces, metrics, logs) plus a Grafana Cloud overlay that adds 0.5% tail sampling. A six-service Docker Compose stack — collector, Tempo, Loki, Prometheus, Grafana, renderer — gives a complete local environment. ➡️ **[View full Configuration Reference](./05-configuration-reference.md)** @@ -168,20 +182,31 @@ OpenTelemetry Collector configurations are provided for development and producti ## 6. Implementation Phases -The implementation spans 13 weeks across 8 phases: +The plan was originally scoped at **13 weeks across 8 phases** — the table below +is that original scope. As delivered it grew to **11 phases through week 20**; +Phases 9-11 were added after the original plan was written. See +[06-implementation-phases.md §6.12.6](./06-implementation-phases.md) for the +authoritative per-phase status, and treat the eight rows below as the +originally-planned subset rather than the current timeline: -| Phase | Duration | Focus | Key Deliverables | -| ----- | ----------- | --------------------- | ----------------------------------------------------------- | -| 1 | Weeks 1-2 | Core Infrastructure | SDK integration, Telemetry interface, Configuration | -| 2 | Weeks 3-4 | RPC Tracing | HTTP context extraction, Handler instrumentation | -| 3 | Weeks 5-6 | Transaction Tracing | Protocol Buffer context, Relay propagation | -| 4 | Weeks 7-8 | Consensus Tracing | Round spans, Proposal/validation tracing | -| 5 | Week 9 | Documentation | Runbook, Dashboards, Training | -| 6 | Week 10 | StatsD Metrics Bridge | OTel Collector StatsD receiver, 3 Grafana dashboards | -| 7 | Weeks 11-12 | Native OTel Metrics | OTelCollector impl, OTLP metrics export, StatsD deprecation | -| 8 | Week 13 | Log-Trace Correlation | trace_id in logs, Loki ingestion, Tempo↔Loki linking | +| Phase | Duration | Focus | Key Deliverables | +| ----- | ----------- | --------------------- | --------------------------------------------------------- | +| 1 | Weeks 1-2 | Core Infrastructure | SDK integration, Telemetry interface, Configuration | +| 2 | Weeks 3-4 | RPC Tracing | HTTP context extraction, Handler instrumentation | +| 3 | Weeks 5-6 | Transaction Tracing | Protocol Buffer context, Relay propagation | +| 4 | Weeks 7-8 | Consensus Tracing | Round spans, Proposal/validation tracing | +| 5 | Week 9 | Documentation | Runbook, Dashboards, Training | +| 6 | Week 10 | StatsD Metrics Bridge | OTel Collector StatsD receiver, 3 Grafana dashboards | +| 7 | Weeks 11-12 | Native OTel Metrics | OTelCollector impl, OTLP metrics export (StatsD retained) | +| 8 | Week 13 | Log-Trace Correlation | trace_id in logs, Loki ingestion, Tempo↔Loki linking | -**Total Effort**: 65.1 developer-days with 2 developers +Delivered beyond the original scope: **Phase 9** (weeks 14-15, internal metric +instrumentation gap fill), **Phase 10** (weeks 16-17, synthetic workload +generation and telemetry validation) and **Phase 11** (weeks 18-20, third-party +data-collection pipelines). + +**Total Effort**: 65.1 developer-days with 2 developers, for the eight +originally-planned phases only. ➡️ **[View full Implementation Phases](./06-implementation-phases.md)** @@ -191,9 +216,9 @@ The implementation spans 13 weeks across 8 phases: > **APM** = Application Performance Monitoring | **GCS** = Google Cloud Storage -Grafana Tempo is recommended for all environments due to its cost-effectiveness and Grafana integration, while Elastic APM is ideal for organizations with existing Elastic infrastructure. +Grafana Tempo is recommended for all environments due to its cost-effectiveness and Grafana integration, and it is the only backend this repo provisions. Elastic APM remains a reasonable choice for organizations with existing Elastic infrastructure, but nothing here configures it. -The recommended production architecture uses a gateway collector pattern with regional collectors performing tail-based sampling, routing traces to multiple backends (Tempo for primary storage, Elastic for log correlation, S3/GCS for long-term archive). +The recommended production architecture uses a gateway collector pattern with regional collectors performing tail-based sampling, routing traces to multiple backends (Tempo for primary storage, Elastic for log correlation, S3/GCS for long-term archive). Note that several subsections of doc 7 predate the shipped dashboards and alert rules and are marked superseded in place, pointing at [09-data-collection-reference.md](./09-data-collection-reference.md) and `docs/telemetry-runbook.md`. ➡️ **[View Observability Backend Recommendations](./07-observability-backends.md)** @@ -209,7 +234,7 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe ## 9. Data Collection Reference -A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo search guides and Prometheus query examples. +A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld: the OpenTelemetry span inventory with per-span attributes, the `beast::insight` and native `XRPL_METRIC_*` instruments (gauges, counters, histograms, overlay traffic), the SpanMetrics-derived Prometheus metrics, and the **15** Grafana dashboards. Includes Tempo search guides and Prometheus query examples. Consult that document rather than this index for any count — it tracks the code, this summary does not. ➡️ **[View Data Collection Reference](./09-data-collection-reference.md)** diff --git a/OpenTelemetryPlan/Phase10_taskList.md b/OpenTelemetryPlan/Phase10_taskList.md index 7022652b9e..e1b9450ae6 100644 --- a/OpenTelemetryPlan/Phase10_taskList.md +++ b/OpenTelemetryPlan/Phase10_taskList.md @@ -22,10 +22,13 @@ Before Phases 1-9 can be considered production-ready, we need proof that: -1. All required spans fire with correct attributes under real transaction workloads +1. Every emitted span fires with its required attributes under real transaction + workloads — the harness derives the span and attribute totals from + `expected_spans.json`, so no fixed "16 spans / 22 attributes" figure applies 2. All 255+ StatsD metrics + ~50 Phase 9 metrics appear in Prometheus with non-zero values 3. Log-trace correlation (Phase 8) produces clickable trace_id links in Loki -4. All 10 Grafana dashboards render meaningful data (no empty panels) +4. The 14 harness-asserted Grafana dashboards render meaningful data (no empty + panels); 15 are on disk 5. Performance overhead stays within bounds (< 3% CPU, < 5MB memory) 6. The telemetry stack survives sustained load without data loss or queue backpressure @@ -37,25 +40,42 @@ Before Phases 1-9 can be considered production-ready, we need proof that: **What to do**: -- Create `docker/telemetry/docker-compose.workload.yaml`: - - 5 xrpld validator nodes with UNL configured for each other - - All telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel` - - Full OTel stack: Collector, Tempo, Prometheus, Loki, Grafana - - Shared network with service discovery +- Create `docker/telemetry/docker-compose.workload.yaml` — **as shipped this file + holds only the observability backend**: `otel-collector`, `tempo`, + `prometheus`, `loki`, `grafana`. It contains **no xrpld services**. + - Shared network (`workload-net`) with service discovery -- Each node should: - - Generate validator keys at startup - - Configure all 5 nodes in its UNL - - Enable all trace categories including `trace_peer=1` - - Write logs to a file tailed by the OTel Collector filelog receiver +- The 5 validators are **native `xrpld` processes**, not containers. + `docker/telemetry/workload/run-full-validation.sh` (`NUM_NODES=5`) generates + keys, writes a per-node `xrpld.cfg`, and launches each node on + `127.0.0.1` with sequential RPC / WS / peer ports. Each node: + - Gets its validator key from `generate-validator-keys.sh` + - Lists the other 4 nodes in `ips_fixed` + - Has all telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel` + - Enables all trace categories including `trace_peer=1` + - Writes logs to a file tailed by the OTel Collector filelog receiver -- Include a `Makefile` target: `make telemetry-workload-up` / `make telemetry-workload-down` +- ❌ **`make telemetry-workload-up` / `make telemetry-workload-down` were never + implemented.** There is no `Makefile` anywhere in the repository. The entry + point is `run-full-validation.sh` (with `--profile`, `--nodes`, + `--skip-loki`, `--skip-regression`, `--with-benchmark`). The node-count flag is + spelled `--nodes`, **not** `--num-nodes` — `run-full-validation.sh:80` (usage) + and `:100` (the `case` arm). `NUM_NODES` is the internal shell variable it + assigns to. **Key files**: -- New: `docker/telemetry/docker-compose.workload.yaml` +- New: `docker/telemetry/docker-compose.workload.yaml` (backend only) - New: `docker/telemetry/workload/generate-validator-keys.sh` -- New: `docker/telemetry/workload/xrpld-validator.cfg.template` +- New: `docker/telemetry/workload/run-full-validation.sh` — writes each node's + cfg **inline** via a heredoc at `run-full-validation.sh:242` + (`cat >"$NODE_DIR/xrpld.cfg" < `rpc.process` -> `rpc.command.*` + - WebSocket: `rpc.ws_message` -> `rpc.command.*` — **there is no + `rpc.process` on the WS path**. `rpc.process` is created only in + `ServerHandler::processRequest()` (`ServerHandler.cpp:705`), reached from + `processSession(Session, coro)`, i.e. HTTP only. Under WS-only load + `rpc.process` never appears, and `rpc.command.*` parents directly to + `rpc.ws_message`. - Assert span durations are reasonable (> 0, < 60s) **Metric validation** (queries Prometheus API): - - Assert all SpanMetrics-derived metrics are non-zero: `traces_span_metrics_calls_total`, `traces_span_metrics_duration_milliseconds_bucket` - - Assert all StatsD metrics are non-zero: `xrpld_LedgerMaster_Validated_Ledger_Age`, `xrpld_Peer_Finder_Active_*`, etc. - - Assert all Phase 9 metrics are non-zero: `xrpld_nodestore_*`, `xrpld_cache_*`, `xrpld_txq_*`, `xrpld_rpc_method_*`, `xrpld_object_count`, `xrpld_load_factor*` + - Assert all SpanMetrics-derived metrics are non-zero: `span_calls_total`, + `span_duration_milliseconds_bucket` (the connector's `namespace` is `span`, + not `traces_span_metrics` — `otel-collector-config.yaml:113-114`) + - Assert the insight-sourced metrics are non-zero: `ledgermaster_validated_ledger_age`, + `peer_finder_active_{inbound,outbound}_peers`, etc. — all lowercase, no + `xrpld_` prefix (`77f35c03db` removed the prefix and lowercased names) + - Assert all Phase 9 metrics are non-zero: `nodestore_state`, `cache_metrics`, + `txq_metrics`, `rpc_method_{started,finished,errored}_total`, `object_count`, + `load_factor_metrics` - Assert metric label cardinality is within bounds **Log-trace correlation validation** (queries Loki API): @@ -142,7 +176,9 @@ Before Phases 1-9 can be considered production-ready, we need proof that: - Assert Grafana derived field links are functional **Dashboard validation**: - - For each of the 10 Grafana dashboards, query the dashboard API and assert no panels show "No data" + - For each dashboard, query the dashboard API and assert no panels show "No + data". There are **15 dashboards on disk**; the harness asserts **14** — + `log-derived-insights` is provisioned but unasserted. - Output: JSON report with pass/fail per check, suitable for CI. @@ -234,17 +270,28 @@ Before Phases 1-9 can be considered production-ready, we need proof that: ## Exit Criteria — Delivered in PR #6519 -- [x] Multi-node validator cluster starts and reaches consensus +- [x] 5-node validator cluster starts and reaches consensus — as native `xrpld` + processes driven by `run-full-validation.sh` (`NUM_NODES=5`), not from + docker-compose - [x] RPC load generator fires all traced RPC commands at configurable rates - [x] Transaction submitter generates 6+ transaction types at configurable TPS -- [x] Validation suite confirms all required spans, attributes, and metrics -- [x] Log-trace correlation validated end-to-end (Loki ↔ Tempo) -- [x] Grafana dashboards render data (no empty panels) -- [x] Overhead benchmark (`benchmark.sh`) measures telemetry-off vs telemetry-on deltas +- [x] Validation suite confirms the full span / attribute / metric inventory + (totals computed dynamically from `expected_spans.json` / + `expected_metrics.json`) +- [x] Log-trace correlation validated end-to-end (Loki <-> Tempo) — implemented + and passing locally, but CI runs with `--skip-loki`, so it is not gated +- [x] All 14 harness-asserted Grafana dashboards render data (no empty panels); + 15 on disk, `log-derived-insights` unasserted +- [x] Overhead benchmark (`benchmark.sh`) measures telemetry-off vs telemetry-on + deltas +- [ ] Benchmark shows < 3% CPU overhead, < 5MB memory overhead — needs a + measured run - [x] CI workflow runs validation on telemetry branch changes - [x] Validation report output is CI-parseable (JSON with exit codes) -- [x] OTel-driven regression gate captures per-span/per-RPC/per-job timings from - Prometheus and compares against a committed baseline +- [x] OTel-driven regression gate captures per-span and per-job timings from + Prometheus and compares against a committed baseline. Per-RPC timings are + **not** gated: `regression-metrics.json` defines only `spans` and + `job_queue` groups (FU-4). ## Follow-up Work (tracked in separate PRs) @@ -253,6 +300,9 @@ Before Phases 1-9 can be considered production-ready, we need proof that: requires a manual baseline-refresh PR. - [ ] FU-4: Replace the proxy measurements in `benchmark.sh` (wall-clock curl p99, ledger-cadence-as-TPS, ledger-cadence-as-consensus-p95) with - PromQL quantile queries from the same pipeline the regression gate uses. + PromQL quantile queries from the same pipeline the regression gate uses, + and add an `rpc_methods` group to `regression-metrics.json` plus a + `defaults.rpc_method` block to `regression-thresholds.json` (without both, + any `rpc.*` metric resolves to "no threshold configured" and never gates). - [ ] FU-6: Grafana dashboard plotting historical baseline values keyed by commit SHA, for triaging noisy regressions. diff --git a/OpenTelemetryPlan/Phase11_taskList.md b/OpenTelemetryPlan/Phase11_taskList.md index 7429063984..0630417e1c 100644 --- a/OpenTelemetryPlan/Phase11_taskList.md +++ b/OpenTelemetryPlan/Phase11_taskList.md @@ -1,6 +1,17 @@ # Phase 11: Third-Party Data Collection Pipelines — Task List -> **Status**: Future Enhancement +> **Status**: Not started — 0 of 13 tasks complete (`grep -c '^## Task 11\.'` = 13: +> Tasks 11.1 through 11.13). Verified against the tree: +> no `.go` files exist anywhere, `docker/telemetry/otel-rippled-receiver/` does +> not exist, `docker/telemetry/prometheus/` does not exist (so no +> `prometheus/rippled-alerts.yml`), and no `network-topology` / `dex-amm` +> dashboards are present under `docker/telemetry/grafana/dashboards/`. **No Phase 11 work has +> been done, so no task box below may be ticked.** +> +> One **prerequisite** box is ticked, and only one: Task 11.12's +> "`state_tracking` gauge implemented (Task 7.12)". That is an upstream +> dependency satisfied by Phase 7/9 code, not Phase 11 work — see the citation +> there. > > **Goal**: Build a custom OTel Collector receiver that periodically polls xrpld's admin RPCs and exports structured metrics for external consumers — making all XRPL health, validator, peer, fee, and DEX data available as Prometheus/OTLP metrics without xrpld code changes. > @@ -287,7 +298,35 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h ## Task 11.8: Prometheus Alerting Rules -**Objective**: Create production-ready alerting rules for the metrics exported by this receiver. +**Objective**: Create production-ready alerting rules for the `xrpl_*` metrics +exported by this receiver. + +> **Scope note — do not duplicate Phase 9.** Phase 9 already ships provisioned +> **Grafana** alerting at +> `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` +> — 13 rules in 5 groups, 2 contact points (`xrpld-default` Slack, +> `xrpld-critical` Slack + email), and a nested notification policy keyed on +> `severity = critical`. Four of the rules below overlap it: +> +> | Rule here | Addressed by (Phase 9) | Coverage | +> | ------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +> | `XRPLServerNotFull` | `NodeNotFull` (group `xrpld-node-state`) | Full | +> | `XRPLLedgerStale` | `ValidatedLedgerStale` (group `xrpld-consensus`) | **Partial** — Phase 9: `ledgermaster_validated_ledger_age > 60` for 5m; the external shape is `> 30` for 1m | +> | `XRPLHighIOLatency` | `NodeStoreIOLatencyHigh` (group `xrpld-jobqueue`) | **Partial** — Phase 9: p95 of `ios_latency_milliseconds_bucket` **> 1000 ms for 10m**; external: **> 50 for 1m** | +> | `XRPLStateFlapping` | `NodeStateFlapping` (group `xrpld-node-state`) | Full | +> +> The remaining 8 (`XRPLAmendmentBlocked`, `XRPLNoPeers`, +> `XRPLUnsupportedAmendmentMajority`, `XRPLLowPeerCount`, `XRPLHighLoadFactor`, +> `XRPLSlowConsensus`, `XRPLValidatorListExpiring`, `XRPLClockDrift`) are +> genuinely new. Note the two sets watch different metric surfaces — the Phase 9 +> rules fire on xrpld's own OTLP metrics, these on the receiver's `xrpl_*` +> metrics — so if both are kept, dedupe the notification policy to avoid +> double-paging on the same underlying condition. +> +> `docker/telemetry/prometheus/` does not exist today. Prefer extending the +> Phase 9 Grafana provisioning tree over introducing a second, Prometheus-native +> alerting mechanism; if a `prometheus/` tree is added anyway, say explicitly in +> its header which alerts it owns. **What to do**: @@ -360,9 +399,22 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h **Objective**: Create 4 new dashboards for the data exported by the receiver. +> **UID COLLISION — pick a different uid.** Phase 9 already ships +> `docker/telemetry/grafana/dashboards/validator-health.json` with +> **uid `validator-health`** (17 panels, backed by xrpld's own +> `validation_agreement` / `validator_health` / `state_tracking` OTLP metrics). +> Provisioning a second dashboard with the same uid makes Grafana overwrite one +> with the other — whichever the provisioner loads last wins, silently. Use a +> distinct uid such as `validator-health-external` (and a distinct filename), the +> same way this task already disambiguates Fee Market as +> `xrpld-fee-market-external` against Phase 9's `fee-market`. Also check +> `peer-quality`, `fee-market`, `job-queue` and `node-health` before adding any +> further uid. + **What to do**: -- **Validator Health** (`validator-health`): +- **Validator Health** (`validator-health-external` — **not** `validator-health`, + see the collision note above): - Server state timeline, state duration breakdown - Proposer count trend, converge time trend, validation quorum - Validator list expiration countdown @@ -386,10 +438,16 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h **Key files**: -- New: `docker/telemetry/grafana/dashboards/rippled-validator-health.json` -- New: `docker/telemetry/grafana/dashboards/rippled-network-topology.json` -- New: `docker/telemetry/grafana/dashboards/rippled-fee-market-external.json` -- New: `docker/telemetry/grafana/dashboards/rippled-dex-amm.json` +- New: `docker/telemetry/grafana/dashboards/validator-health-external.json` + (**must not** reuse Phase 9's `validator-health.json` / uid `validator-health`) +- New: `docker/telemetry/grafana/dashboards/network-topology.json` +- New: `docker/telemetry/grafana/dashboards/fee-market-external.json` + (Phase 9 owns `fee-market.json` / uid `fee-market`) +- New: `docker/telemetry/grafana/dashboards/dex-amm.json` + +> Filenames drop the legacy `dashboards/rippled-*` prefix: `145b1469d6` and +> `25868f2740` renamed every dashboard to bare names with bare uids, so no +> `dashboards/rippled-*.json` path exists in the tree. --- @@ -446,20 +504,55 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h > **Upstream**: Phase 7 Tasks 7.9-7.16 (metrics), Phase 9 Tasks 9.11-9.13 (dashboards). > **Downstream**: None — terminal task in the parity chain. -**Objective**: Add Grafana alerting rules for the Phase 7+ parity metrics (validation agreement, validator health, peer quality, state tracking, ledger economy). These complement Task 11.8's `xrpl_*` alerts by covering the `xrpld_*` internal metrics. +**Objective**: Add Grafana alerting rules for the Phase 7+ parity metrics (validation agreement, validator health, peer quality, state tracking, ledger economy). These complement Task 11.8's `xrpl_*` alerts by covering the internal metrics. + +> **4 of the 18 are addressed by Phase 9** — 2 fully, 2 only partially. Extend, +> do not blindly re-create: +> +> | Rule here | Addressed by (Phase 9) | Coverage | +> | ------------------ | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +> | Unhealthy State | `NodeNotFull` (group `xrpld-node-state`) | Full | +> | High IO Latency | `NodeStoreIOLatencyHigh` (group `xrpld-jobqueue`, p95 of `ios_latency_milliseconds_bucket`) | **Partial** — Phase 9 fires at p95 **> 1000 ms for 10m**; the rule below wants **> 50 for 1m** (20× tighter) | +> | Job Queue Overflow | `JobQueueTxOverflow` (group `xrpld-jobqueue`, `jq_trans_overflow_total`) | Full | +> | Stale Ledger | `ValidatedLedgerStale` (group `xrpld-consensus`, `ledgermaster_validated_ledger_age`) | **Partial** — different metric: Phase 9 uses `ledgermaster_validated_ledger_age > 60` for 5m; the rule below uses `ledger_economy{metric="ledger_age_seconds"} > 30` for 1m | +> +> The two **Partial** rows are not closed. Either re-baseline the Phase 9 +> thresholds or ship the tighter variants here — do not skip them as duplicates. +> +> Remaining open work is **14 rules**, of which **3** (CPU High, Memory Critical, +> Disk Warning) need `node_exporter`, which is not in the stack. Nothing else is +> blocked: "Not Proposing" used to be listed as blocked on an unimplemented +> `state_tracking` gauge, but that gauge **ships** — see the Exit Criteria note +> below. +> +> **Metric-name translation.** Names carry **no** `xrpld_` prefix +> (`77f35c03db`), so as a rule of thumb read every `xrpld_` below as plain +> ``. **Two shapes do not follow that rule:** +> +> - **Multiplexed observable gauges.** Many readings are a `metric` **label +> value** on a shared instrument, not a metric name. `xrpld_txq_count` is +> `txq_metrics{metric="txq_count"}`; likewise `load_factor_metrics{…}`, +> `nodestore_state{…}`, `cache_metrics{…}`. The rows below that already use the +> `{metric="…"}` form (`state_tracking`, `validator_health`, +> `validation_agreement`, `server_info`, `peer_quality`, `load_factor_metrics`, +> `ledger_economy`) are correct; only drop the prefix on those. +> - **Unit-suffixed histograms** from `beast::insight`. `OTelCollectorImp` appends +> the unit, so `xrpld_ios_latency_bucket` is really +> `ios_latency_milliseconds_bucket` — the spelling used by +> `node-health.json:577` and `ledger-data-sync.json:1353`. **Critical Group** (8 rules, eval interval 10s): -| Rule | Condition | For | -| ------------------- | ------------------------------------------------------------- | --- | -| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s | -| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s | -| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s | -| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m | -| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h | -| High IO Latency | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50` | 1m | -| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m | -| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m | +| Rule | Condition | For | +| ------------------- | ---------------------------------------------------------------- | --- | +| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s | +| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s | +| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s | +| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m | +| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h | +| High IO Latency | `histogram_quantile(0.95, ios_latency_milliseconds_bucket) > 50` | 1m | +| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m | +| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m | **Network Group** (3 rules, eval interval 10s): @@ -481,19 +574,44 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h | TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m | | Stale Ledger | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30` | 1m | -**Notification channel templates**: Email/SMTP, Discord, Slack, PagerDuty. +**Notification channel templates**: Slack and Email/SMTP already ship in Phase +9's `contactpoints.yaml` (`xrpld-default`, `xrpld-critical`). Discord and +PagerDuty templates remain open. -**Key files**: +**Key files** — extend the **Phase 9** provisioning tree. The +`docker/telemetry/grafana/alerting/` directory named in the original spec has +never existed in any commit; the real location is +`docker/telemetry/grafana/provisioning/alerting/`: -- New/extend: `docker/telemetry/grafana/alerting/alert-rules-parity.yaml` -- New: `docker/telemetry/grafana/alerting/contact-points.yaml` (template configs) -- New: `docker/telemetry/grafana/alerting/notification-policies.yaml` +- Extend: `docker/telemetry/grafana/provisioning/alerting/rules.yaml` (add groups + alongside the existing `xrpld-consensus`, `xrpld-validator`, `xrpld-jobqueue`, + `xrpld-node-state`, `xrpld-overlay`) +- Extend: `docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml` + (add Discord / PagerDuty receivers) +- Extend: `docker/telemetry/grafana/provisioning/alerting/policies.yaml` + (add routes; the root route and the `severity = critical` child already exist) **Exit Criteria**: -- [ ] All 18 rules evaluate without errors in Grafana alerting UI +- [ ] The 14 not-yet-shipped rules evaluate without errors in Grafana alerting UI +- [ ] The 2 rules **fully** covered by Phase 9 (Unhealthy State, Job Queue + Overflow) are not duplicated; the 2 **partially** covered ones (High IO + Latency, Stale Ledger) are either re-baselined on the Phase 9 rule or shipped + as tighter variants — decision recorded either way - [ ] Critical rules fire within expected timeframe when conditions are met - [ ] Notification channel templates are documented (not hard-coded to any service) +- [ ] `node_exporter` decision recorded for the 3 host-level rules (CPU, memory, disk) +- [x] `state_tracking` gauge implemented (Task 7.12) before adding "Not Proposing" + — **prerequisite met upstream**, not Phase 11 work. + `MetricsRegistry::registerStateTrackingGauge()` + (`src/xrpld/telemetry/MetricsRegistry.cpp:1461-1510`) creates + `CreateDoubleObservableGauge("state_tracking", "Node state and mode tracking")` + at `:1466` and observes `state_value` (`:1497`) and + `time_in_current_state_seconds` (`:1502`). Already queried by + `validator-health.json:765,971` and `ledger-data-sync.json:869`, and + documented in + [09-data-collection-reference.md](./09-data-collection-reference.md) + § State Tracking. "Not Proposing" can be written now. --- @@ -533,12 +651,14 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h - [ ] Custom OTel Collector receiver builds and starts without errors - [ ] All `xrpl_*` metrics from server_info, get_counts, peers, validators, fee appear in Prometheus - [ ] Metrics update at configured poll interval (default 30s) -- [ ] 4 new Grafana dashboards operational with data +- [ ] 4 new Grafana dashboards operational with data, none reusing a Phase 9 uid + (`validator-health`, `peer-quality`, `fee-market`, `job-queue`, `node-health`) - [ ] Prometheus alerting rules fire correctly for simulated failure conditions - [ ] DEX/AMM collector works when configured (optional — not required for base exit criteria) - [ ] Phase 10 validation suite passes with receiver metrics included - [ ] Receiver handles xrpld restart/unavailability gracefully (no crash, logs warning, retries) - [ ] Documentation complete: receiver README, metric reference, alerting playbook - [ ] Go receiver has unit tests with >80% coverage -- [ ] 18 Grafana alert rules for Phase 7+ parity metrics evaluate correctly (Task 11.12) +- [ ] The 14 not-yet-shipped Grafana alert rules for Phase 7+ parity metrics + evaluate correctly (Task 11.12); the other 4 of the 18 already ship in Phase 9 - [ ] Dual-datasource architecture documented with trade-offs (Task 11.13) diff --git a/OpenTelemetryPlan/Phase2_taskList.md b/OpenTelemetryPlan/Phase2_taskList.md index 74d13d54c3..1f93ff7265 100644 --- a/OpenTelemetryPlan/Phase2_taskList.md +++ b/OpenTelemetryPlan/Phase2_taskList.md @@ -8,11 +8,11 @@ ### Related Plan Documents -| Document | Relevance | -| ------------------------------------------------------------ | ------------------------------------------------------------- | -| [04-code-samples.md](./04-code-samples.md) | TraceContextPropagator (§4.4.2), RPC instrumentation (§4.5.3) | -| [02-design-decisions.md](./02-design-decisions.md) | W3C Trace Context (§2.5), span attributes (§2.4.2) | -| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 2 tasks (§6.3), definition of done (§6.11.2) | +| Document | Relevance | +| ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| [03-implementation-strategy.md](./03-implementation-strategy.md) | Code structure and instrumentation patterns (replaces the deleted `04-code-samples.md` §4.4.2 / §4.5.3, removed by `d6450631bf`) | +| [02-design-decisions.md](./02-design-decisions.md) | W3C Trace Context (§2.5), span attributes (§2.4.2) | +| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 2 tasks (§6.3), definition of done (§6.11.2) | --- @@ -111,7 +111,7 @@ These can be added later if dashboard queries specifically need them. The node h **Verification Checklist**: - [ ] `conan install . --build=missing -o telemetry=True` succeeds -- [ ] `cmake --preset default -Dtelemetry=ON` configures correctly +- [ ] `cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -Dtelemetry=ON ..` configures correctly (there is no `default` preset; Conan writes `conan-release`) - [ ] Build succeeds with telemetry ON - [ ] Build succeeds with telemetry OFF - [ ] Existing tests pass with telemetry ON diff --git a/OpenTelemetryPlan/Phase3_taskList.md b/OpenTelemetryPlan/Phase3_taskList.md index c5d3c95251..55e0a9ed64 100644 --- a/OpenTelemetryPlan/Phase3_taskList.md +++ b/OpenTelemetryPlan/Phase3_taskList.md @@ -8,12 +8,12 @@ ### Related Plan Documents -| Document | Relevance | -| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | -| [04-code-samples.md](./04-code-samples.md) | TraceContext protobuf (§4.4.1), PeerImp instrumentation (§4.5.1), context serialization (§4.4.2) | -| [01-architecture-analysis.md](./01-architecture-analysis.md) | Transaction flow (§1.3), key trace points (§1.6) | -| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 3 tasks (§6.4), definition of done (§6.11.3) | -| [02-design-decisions.md](./02-design-decisions.md) | Context propagation design (§2.5), attribute schema (§2.4.3) | +| Document | Relevance | +| ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) | Authoritative protocol span-flow reference — replaces the deleted `04-code-samples.md` (TraceContext protobuf §4.4.1, PeerImp instrumentation §4.5.1, context serialization §4.4.2), removed by `d6450631bf` | +| [01-architecture-analysis.md](./01-architecture-analysis.md) | Transaction flow (§1.3), key trace points (§1.6) | +| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 3 tasks (§6.4), definition of done (§6.11.3) | +| [02-design-decisions.md](./02-design-decisions.md) | Context propagation design (§2.5), attribute schema (§2.4.3) | --- @@ -47,7 +47,9 @@ **Reference**: -- [04-code-samples.md §4.4.1](./04-code-samples.md) — TraceContext message definition +- `04-code-samples.md` §4.4.1 (TraceContext message definition) was deleted by + `d6450631bf`; the live definition is `include/xrpl/proto/xrpl.proto:101` + (`message TraceContext`), attached as field `1001` on the relevant messages - [02-design-decisions.md §2.5.2](./02-design-decisions.md) — Protocol buffer context propagation design --- @@ -75,7 +77,13 @@ **Reference**: -- [04-code-samples.md §4.4.2](./04-code-samples.md) — Full extract/inject implementation +- `04-code-samples.md` §4.4.2 (full extract/inject implementation) was deleted by + `d6450631bf`. As shipped there is **no** + `src/libxrpl/telemetry/TraceContextPropagator.cpp`; extract/inject live on + `SpanGuard` (`include/xrpl/telemetry/SpanGuard.h:467` extract, `:480-491` + `injectCurrentContextToProtobuf`, implemented in + `src/libxrpl/telemetry/SpanGuard.cpp`) with the protocol-layer wrappers in + `src/xrpld/telemetry/PropagationHelpers.h:52` (`injectSpanContext`) --- @@ -110,7 +118,9 @@ **Reference**: -- [04-code-samples.md §4.5.1](./04-code-samples.md) — Full PeerImp instrumentation example +- [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) + — the authoritative `tx.receive` / relay span-flow reference; replaces + `04-code-samples.md` §4.5.1, deleted by `d6450631bf` - [01-architecture-analysis.md §1.3](./01-architecture-analysis.md) — Transaction flow diagram - [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — tx.receive trace point @@ -231,7 +241,10 @@ design. **Reference**: - [02-design-decisions.md §2.5](./02-design-decisions.md) — Context propagation design -- [04-code-samples.md §4.5.1](./04-code-samples.md) — Relay context injection pattern +- Relay context injection pattern: `04-code-samples.md` §4.5.1 was deleted by + `d6450631bf`; the live pattern is `injectSpanContext()` in + `src/xrpld/telemetry/PropagationHelpers.h:52`, with the flow documented in + [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) --- diff --git a/OpenTelemetryPlan/Phase4_taskList.md b/OpenTelemetryPlan/Phase4_taskList.md index e83a16262e..e5e380a9d6 100644 --- a/OpenTelemetryPlan/Phase4_taskList.md +++ b/OpenTelemetryPlan/Phase4_taskList.md @@ -6,21 +6,53 @@ > > **Branch**: `pratik/otel-phase4-consensus-tracing` (from `pratik/otel-phase3-tx-tracing`) -> **Note on attribute names**: the `xrpl..` keys shown below are -> written in the older dotted form for readability — it mirrors how the fully -> qualified attribute reads in a Tempo trace view. The implemented keys follow -> the convention in [CONTRIBUTING.md](../CONTRIBUTING.md#telemetry-span-attribute-naming) -> (underscore form, e.g. `consensus_round`, `consensus_mode`); the +> **Note on attribute names**: the `xrpl..` keys that earlier +> revisions of this task list used were **never emitted**. `9e27120a15` removed +> the dotted `xrpl.*` namespace from **span** attributes repo-wide. Falsifiable +> check: `grep -rn 'seg::xrpl' src/ include/` → exactly **2** hits, both +> `include/xrpl/telemetry/SpanNames.h:117-118` (`attr::networkId`, +> `attr::networkType`), and both are **resource** attributes +> (`xrpl.network.id` / `xrpl.network.type`) set on the OTel resource at startup — +> the one place the dotted form is still reserved. No span attribute uses it. +> (Do **not** cite `grep 'makeStr("xrpl\.' src/ include/` → 0 hits as evidence: +> these keys were always composed with `join(seg::…, …)`, never that literal, so +> the grep has returned 0 for the entire history of the file and cannot fail.) +> Those spellings have been corrected in place, so every attribute key below is +> the live one. The mapping that was applied: +> `xrpl.ledger.seq` → `ledger_seq`, `xrpl.consensus.mode` → `consensus_mode`, +> `xrpl.consensus.round` → `consensus_round`, +> `xrpl.consensus.round_id` → `consensus_round_id`, +> `xrpl.consensus.ledger_id` → `consensus_ledger_id`, +> `xrpl.tx.id` → `tx_id`, +> `xrpl.validation.ledger_hash` / `xrpl.peer.validation.ledger_hash` → `ledger_hash`, +> `xrpl.validation.full` / `xrpl.peer.validation.full` → `full_validation`, +> `xrpl.peer.version` → `peer_version`. +> Separately, `19a6c2a306` split the single `trusted` key into +> `proposal_trusted` (on `consensus.proposal.receive` and `peer.proposal.receive`) +> and `validation_trusted` (on `consensus.validation.receive` and +> `peer.validation.receive`). Naming follows +> [CONTRIBUTING.md](../CONTRIBUTING.md#telemetry-span-attribute-naming); the > `*SpanNames.h` constants are the single source of truth. +> +> **Three names in this document are not span attributes at all**: +> +> - `amendment_blocked` — a **metric label value** only: +> `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1216`). +> No span carries it. +> - `server_state` — a **metric label value** only: +> `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1014`). It is also +> an RPC method name. No span carries it. +> - `proposers_validated` — **never implemented** on any span. `proposersValidated` +> exists only as a C++ function/parameter name (`RCLConsensus.cpp:310`); +> `consensus.accept` carries `proposers` instead (see Task 4.8). ### Related Plan Documents -| Document | Relevance | -| ------------------------------------------------------------ | ----------------------------------------------------------- | -| [04-code-samples.md](./04-code-samples.md) | Consensus instrumentation (§4.5.2), consensus span patterns | -| [01-architecture-analysis.md](./01-architecture-analysis.md) | Consensus round flow (§1.4), key trace points (§1.6) | -| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 4 tasks (§6.5), definition of done (§6.11.4) | -| [02-design-decisions.md](./02-design-decisions.md) | Consensus attribute schema (§2.4.4) | +| Document | Relevance | +| ------------------------------------------------------------ | -------------------------------------------------------------------- | +| [01-architecture-analysis.md](./01-architecture-analysis.md) | Consensus round flow (§1.4), key trace points (§1.6) | +| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 4 tasks and exit criteria (§6.5), definition of done (§6.12.4) | +| [02-design-decisions.md](./02-design-decisions.md) | Consensus attribute schema (§2.4.2 → "Consensus Attributes") | --- @@ -34,8 +66,8 @@ - `RCLConsensus::Adaptor::startRoundTracing()` creates `consensus.round` span via `SpanGuard::hashSpan()` (deterministic) or `SpanGuard::span()` (attribute strategy) -- Attributes set: `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, - `xrpl.consensus.mode`, `trace_strategy`, `xrpl.consensus.round_id` +- Attributes set: `consensus_ledger_id`, `ledger_seq`, + `consensus_mode`, `trace_strategy`, `consensus_round_id` - Round span stored as `roundSpan_` member in `RCLConsensus::Adaptor` - `roundSpanContext_` snapshot captured for cross-thread span linking @@ -46,7 +78,9 @@ **Reference**: -- [04-code-samples.md §4.5.2](./04-code-samples.md) — startRound instrumentation example +- `RCLConsensus::Adaptor::startRoundTracing()` — the live startRound + instrumentation (the former `04-code-samples.md` §4.5.2 was deleted by + `d6450631bf`; the code is the reference now) - [01-architecture-analysis.md §1.4](./01-architecture-analysis.md) — Consensus round flow --- @@ -75,7 +109,8 @@ **Reference**: -- [04-code-samples.md §4.5.2](./04-code-samples.md) — phaseTransition instrumentation +- `Consensus.h` — the live phase-transition instrumentation (`04-code-samples.md` + was deleted by `d6450631bf`) --- @@ -89,11 +124,13 @@ - In `Adaptor::propose()`: - Creates `consensus.proposal.send` span via `SpanGuard::span()` - - Sets `xrpl.consensus.round` attribute + - Sets `consensus_round` attribute - In `PeerImp::onMessage(TMProposeSet)`: - Creates `consensus.proposal.receive` span - - Sets `trusted` attribute (bool) + - Sets `proposal_trusted` attribute (bool) — `PeerSpanNames.h:41`, + `ConsensusSpanNames.h:244`; renamed from the original `trusted` by + `19a6c2a306`, and the dotted `xrpl.peer.*` form was dropped by `9e27120a15` **Done here** (cross-node propagation, send + receive): @@ -112,8 +149,11 @@ **Reference**: -- [04-code-samples.md §4.5.2](./04-code-samples.md) — peerProposal instrumentation -- [02-design-decisions.md §2.4.4](./02-design-decisions.md) — Consensus attribute schema +- `PeerImp::onMessage(TMProposeSet)` — the live peerProposal instrumentation + (`04-code-samples.md` was deleted by `d6450631bf`) +- [02-design-decisions.md §2.4.2](./02-design-decisions.md) — Consensus attribute + schema (the "Consensus Attributes" table under "Span Attributes by Category"; + §2.4.4 is the Privacy & Sensitive Data Policy, not the schema) --- @@ -130,12 +170,14 @@ - Uses `SpanGuard::linkedSpan()` to create a follows-from link to the round span - Thread-safe: uses `roundSpanContext_` snapshot (captured on consensus thread, read on jtACCEPT thread) - - Sets `xrpl.ledger.seq` and `proposing` attributes + - Sets `ledger_seq` and `proposing` attributes - In `PeerImp::onMessage(TMValidation)`: - Creates `consensus.validation.receive` span - - Sets `trusted` attribute (bool) - - Sets `xrpl.ledger.seq` attribute + - Sets `validation_trusted` attribute (bool) — `PeerSpanNames.h:42`, + `ConsensusSpanNames.h:245`; renamed from the original `trusted` by + `19a6c2a306`, and the dotted `xrpl.peer.*` form was dropped by `9e27120a15` + - Sets `ledger_seq` attribute **Not implemented** (deferred to Phase 4b — cross-node propagation): @@ -155,9 +197,9 @@ **Implemented attributes** (across various spans): -- `xrpl.ledger.seq` — on `consensus.round`, `consensus.accept.apply` -- `xrpl.consensus.round` — on `consensus.proposal.send` -- `xrpl.consensus.mode` — on `consensus.round`, `consensus.ledger_close` +- `ledger_seq` — on `consensus.round`, `consensus.accept.apply` +- `consensus_round` — on `consensus.proposal.send` +- `consensus_mode` — on `consensus.round`, `consensus.ledger_close` - `proposers` — on `consensus.accept`, `consensus.establish`, `consensus.update_positions` - `converge_percent` — on `consensus.establish`, `consensus.update_positions`, `consensus.check` - `tx_count` — on `consensus.accept.apply` span (in `doAccept()`) @@ -185,7 +227,7 @@ - In `doAccept()` (RCLConsensus.cpp): - Records `tx.included` events on the `consensus.accept.apply` span for each transaction in the accepted set - - Each event includes `xrpl.tx.id` attribute with the transaction hash + - Each event includes `tx_id` attribute with the transaction hash - This links consensus traces to individual transactions **Key modified files**: @@ -225,54 +267,73 @@ **Objective**: Add ledger hash, validation type, and quorum data to consensus validation spans on both send and receive paths. This enables trace-level validation agreement analysis — filter by ledger hash to see which validators agreed for a given ledger. -**Status**: Not implemented. None of the enrichment attributes are set. The `consensus.validation.send` span only has `ledger.seq` and `proposing`. The `consensus.accept` span has `quorum` set to `result.proposers` (not the actual validator quorum from `app_.validators().quorum()`). No `PeerImp.cpp` changes were made. +**Status**: Implemented, except `proposers_validated`. + +- `consensus.validation.send` sets `ledger_seq`, `ledger_hash`, `proposing` and + `full_validation` (`RCLConsensus.cpp:975-981`). +- `peer.validation.receive` sets `ledger_hash` and `full_validation` + (`PeerImp.cpp:2573-2574`). +- `consensus.accept` sets `quorum` from `app_.getValidators().quorum()` + (`RCLConsensus.cpp:516`) — the earlier defect where `quorum` carried + `result.proposers` instead of the real validator quorum is **fixed**. +- Still open: `proposers_validated` on `consensus.accept` — never implemented. + `consensus.accept` already carries `proposers` (`RCLConsensus.cpp:513`), so a + second key for the same value was not added. + +All attribute keys are bare/underscore; the dotted `xrpl.*` forms in the spec +below were never emitted as **span** attributes. Check: +`grep -rn 'seg::xrpl' src/ include/` → 2 hits, both `SpanNames.h:117-118` +resource attributes (`xrpl.network.{id,type}`). See the note at the top of this +document for why the old `makeStr("xrpl\.` grep proved nothing. **What to do**: - Edit `src/xrpld/app/consensus/RCLConsensus.cpp`: - On the `consensus.validation.send` span (in `validate()` / `doAccept()`): - - Add `xrpl.validation.ledger_hash` (string) — the ledger hash being validated - - Add `xrpl.validation.full` (bool) — whether this is a full validation (not partial) + - Add `ledger_hash` (string) — the ledger hash being validated + - Add `full_validation` (bool) — whether this is a full validation (not partial) - On the `consensus.accept` span (in `onAccept()`): - - Add `validation_quorum` (int64) — from `app_.validators().quorum()` - - Add `proposers_validated` (int64) — from `result.proposers` + - Add `quorum` (int64) — from `app_.getValidators().quorum()` ✅ shipped + - Add `proposers_validated` (int64) — from `result.proposers` ❌ never + implemented; `proposers` already carries this value - Edit `src/xrpld/overlay/detail/PeerImp.cpp`: - On the `peer.validation.receive` span: - - Add `xrpl.peer.validation.ledger_hash` (string) — from deserialized `STValidation` object - - Add `xrpl.peer.validation.full` (bool) — from `STValidation` flags + - Add `ledger_hash` (string) — from deserialized `STValidation` object + - Add `full_validation` (bool) — from `STValidation` flags **New span attributes**: -| Span | Attribute | Type | Source | -| --------------------------- | ---------------------------------- | ------ | --------------------------------- | -| `consensus.validation.send` | `xrpl.validation.ledger_hash` | string | Ledger hash from validate() args | -| `consensus.validation.send` | `xrpl.validation.full` | bool | Full vs partial validation | -| `peer.validation.receive` | `xrpl.peer.validation.ledger_hash` | string | From STValidation deserialization | -| `peer.validation.receive` | `xrpl.peer.validation.full` | bool | From STValidation flags | -| `consensus.accept` | `validation_quorum` | int64 | `app_.validators().quorum()` | -| `consensus.accept` | `proposers_validated` | int64 | `result.proposers` | +| Span | Attribute (live name) | Type | Source | Status | +| --------------------------- | --------------------- | ------ | --------------------------------- | ------------------------- | +| `consensus.validation.send` | `ledger_hash` | string | Ledger hash from validate() args | ✅ `RCLConsensus.cpp:977` | +| `consensus.validation.send` | `full_validation` | bool | Full vs partial validation | ✅ `RCLConsensus.cpp:981` | +| `peer.validation.receive` | `ledger_hash` | string | From STValidation deserialization | ✅ `PeerImp.cpp:2573` | +| `peer.validation.receive` | `full_validation` | bool | From STValidation flags | ✅ `PeerImp.cpp:2574` | +| `consensus.accept` | `quorum` | int64 | `app_.getValidators().quorum()` | ✅ `RCLConsensus.cpp:516` | +| `consensus.accept` | `proposers_validated` | int64 | `result.proposers` | ❌ never implemented | **Rationale**: The external dashboard's most valuable feature is validation agreement tracking. By recording the ledger hash on both outgoing and incoming validation spans, we create the raw data for agreement analysis at the trace level. Example Tempo query: ``` -{name="consensus.validation.send"} | xrpl.validation.ledger_hash = "A1B2C3..." +{name="consensus.validation.send" && span.ledger_hash = "A1B2C3..."} ``` Phase 7's `ValidationTracker` builds metric-level aggregation (1h/24h agreement %) on top of this data. -**Key modified files (not yet modified)**: +**Key modified files**: -- `src/xrpld/app/consensus/RCLConsensus.cpp` -- `src/xrpld/overlay/detail/PeerImp.cpp` +- `src/xrpld/app/consensus/RCLConsensus.cpp` (`:516`, `:975-981`) +- `src/xrpld/overlay/detail/PeerImp.cpp` (`:2573-2574`) **Exit Criteria**: - [x] `consensus.validation.send` spans carry `ledger_hash` and `full_validation` -- [ ] `peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full` -- [ ] `consensus.accept` spans carry `validation_quorum` and `proposers_validated` +- [x] `peer.validation.receive` spans carry `ledger_hash` and `full_validation` — `PeerImp.cpp:2573-2574` +- [x] `consensus.accept` spans carry `quorum` — `RCLConsensus.cpp:516` +- [ ] `consensus.accept` spans carry `proposers_validated` — **open**, never implemented - [x] Ledger hash attributes match between send and receive for the same ledger -- [ ] No impact on consensus performance +- [ ] No impact on consensus performance — not measured --- @@ -318,13 +379,13 @@ Phase 7's `ValidationTracker` builds metric-level aggregation (1h/24h agreement ### Implemented Spans -| Span Name | Method | Key Attributes | -| --------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `consensus.proposal.send` | `Adaptor::propose` | `xrpl.consensus.round`, `is_bow_out` | -| `consensus.ledger_close` | `Adaptor::onClose` | `xrpl.ledger.seq`, `xrpl.consensus.mode` | -| `consensus.accept` | `Adaptor::onAccept` | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` | -| `consensus.accept.apply` | `Adaptor::doAccept` | `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `xrpl.ledger.seq`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction` | -| `consensus.validation.send` | `Adaptor::onAccept` (via validate) | `proposing`, `ledger_hash`, `ledger_seq`, `full_validation`, `validation_sign_time` | +| Span Name | Method | Key Attributes | +| --------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `consensus.proposal.send` | `Adaptor::propose` | `consensus_round`, `is_bow_out` | +| `consensus.ledger_close` | `Adaptor::onClose` | `ledger_seq`, `consensus_mode` | +| `consensus.accept` | `Adaptor::onAccept` | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` | +| `consensus.accept.apply` | `Adaptor::doAccept` | `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `ledger_seq`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction` | +| `consensus.validation.send` | `Adaptor::onAccept` (via validate) | `proposing`, `ledger_hash`, `ledger_seq`, `full_validation`, `validation_sign_time` | #### Close Time Attributes (consensus.accept.apply) @@ -342,13 +403,15 @@ driven by `avCT_CONSENSUS_PCT` (75% validator agreement threshold): - **`close_time_vote_bins`** — Number of distinct close-time vote bins from peer proposals. Higher values indicate less agreement among validators. - **`resolution_direction`** — Whether close-time resolution `"increased"` (coarser), `"decreased"` (finer), or stayed `"unchanged"` relative to the previous ledger. -**Exit Criteria** (from [06-implementation-phases.md §6.11.4](./06-implementation-phases.md)): +**Exit Criteria** (from [06-implementation-phases.md §6.5](./06-implementation-phases.md) +— §6.11.4 is the WALK phase, i.e. transaction tracing, and does not carry these +criteria; the Phase 4 definition of done is §6.12.4): - [x] Complete consensus round traces - [x] Phase transitions visible (open, establish, close, accept) - [x] Proposals and validations traced — send and receive; relay deferred to Phase 4b - [x] Close time agreement tracked (per `avCT_CONSENSUS_PCT`) -- [x] No impact on consensus timing +- [ ] No impact on consensus timing — **not measured** - [x] Transaction-consensus correlation (Task 4.6) — `tx.included` events in doAccept - [ ] Validation span enrichment (Task 4.8) — not implemented @@ -386,7 +449,7 @@ consensus round share the same trace_id without P2P context propagation. ### Strategy B — Attribute-Based Correlation -Use normal random trace_id but attach `xrpl.consensus.ledger_id` as an attribute +Use normal random trace_id but attach `consensus_ledger_id` as an attribute on every consensus span. Correlation happens at query time via Tempo/Grafana `by attribute` queries. @@ -423,10 +486,10 @@ In `RCLConsensus::Adaptor::startRound()`: 5. Call `startSpan("consensus.round", parentContext)` so the new span inherits the deterministic trace_id. - If `attribute`: start a normal `consensus.round` span, set - `xrpl.consensus.ledger_id = previousLedger.id()` as attribute. + `consensus_ledger_id = previousLedger.id()` as attribute. -Both strategies always set `xrpl.consensus.round_id` (round number) and -`xrpl.consensus.ledger_id` (previous ledger hash) as attributes. +Both strategies always set `consensus_round_id` (round number) and +`consensus_ledger_id` (previous ledger hash) as attributes. --- @@ -542,7 +605,7 @@ spans in `Consensus.h`. - Reads `consensus_trace_strategy` via `app_.getTelemetry().getConsensusTraceStrategy()` - **Deterministic**: uses `SpanGuard::hashSpan()` with `prevLgr.id()` data - **Attribute**: uses `SpanGuard::span(TraceCategory::Consensus, seg::consensus, "round")` - - Sets attributes: `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy`, `xrpl.consensus.round_id` + - Sets attributes: `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id` - Captures `roundSpanContext_` snapshot for cross-thread span linking - Saves `prevRoundContext_` from previous round for follows-from links @@ -811,7 +874,7 @@ and OFF, and don't affect consensus timing. | Span Name | Location | Key Attributes (actually set) | | ---------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | -| `consensus.round` | `RCLConsensus.cpp` | `xrpl.consensus.round_id`, `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy` | +| `consensus.round` | `RCLConsensus.cpp` | `consensus_round_id`, `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy` | | `consensus.establish` | `Consensus.h` | `converge_percent`, `establish_count`, `proposers` | | `consensus.update_positions` | `Consensus.h` | `converge_percent`, `proposers`, `have_close_time_consensus`, `close_time_threshold`, `disputes_count`, `avalanche_threshold` | | `consensus.check` | `Consensus.h` | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `consensus_result` | @@ -819,17 +882,17 @@ and OFF, and don't affect consensus timing. ### New Events (Phase 4a) -| Event Name | Parent Span | Attributes (actually set) | -| ----------------- | ---------------------------- | ---------------------------------------------------------------- | -| `dispute.resolve` | `consensus.update_positions` | `xrpl.tx.id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | -| `tx.included` | `consensus.accept.apply` | `xrpl.tx.id` | +| Event Name | Parent Span | Attributes (actually set) | +| ----------------- | ---------------------------- | ----------------------------------------------------------- | +| `dispute.resolve` | `consensus.update_positions` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | +| `tx.included` | `consensus.accept.apply` | `tx_id` | ### New Attributes (Phase 4a) ```cpp // Round-level (on consensus.round) — ALL IMPLEMENTED -"xrpl.consensus.round_id" = int64 // Consensus round number -"xrpl.consensus.ledger_id" = string // previousLedger.id() hash +"consensus_round_id" = int64 // Consensus round number +"consensus_ledger_id" = string // previousLedger.id() hash "trace_strategy" = string // "deterministic" or "attribute" // Establish-level — IMPLEMENTED @@ -877,9 +940,12 @@ and OFF, and don't affect consensus timing. - **No `getTelemetry()` adaptor method**: `SpanGuard::span()` is a static factory that internally checks telemetry state, so `Consensus.h` doesn't need adaptor access for span creation. Only `RCLConsensus::Adaptor` accesses `app_.getTelemetry()` directly. -- **Config validation**: `consensus_trace_strategy` is validated to be either - `"deterministic"` or `"attribute"`, falling back to `"deterministic"` for - unrecognised values. +- **No config validation**: `consensus_trace_strategy` is **not** validated. + `TelemetryConfig.cpp:155-156` copies the raw string through, and the only + comparison in the code is `strategy == "attribute"` (`RCLConsensus.cpp:1296`). + Any unrecognised value — including a typo — silently takes the deterministic + branch, with no log warning. The effective fallback is correct; the absence of + a diagnostic is a known gap. - **Plan deviation**: `roundSpan_` is stored in `RCLConsensus::Adaptor` (not `Consensus.h`) because the adaptor has access to telemetry config and can implement the deterministic trace ID strategy. `establishSpan_` is correctly diff --git a/OpenTelemetryPlan/Phase7_taskList.md b/OpenTelemetryPlan/Phase7_taskList.md index e22c4b37c9..be90294ff8 100644 --- a/OpenTelemetryPlan/Phase7_taskList.md +++ b/OpenTelemetryPlan/Phase7_taskList.md @@ -75,7 +75,8 @@ - Match existing telemetry code style from `src/libxrpl/telemetry/Telemetry.cpp` - Use RAII for MeterProvider lifecycle (shutdown on destructor) -**Reference**: [04-code-samples.md](./04-code-samples.md) — code style and patterns +**Reference**: [03-implementation-strategy.md](./03-implementation-strategy.md) — +code style and patterns (`04-code-samples.md` was deleted by `d6450631bf`) --- diff --git a/OpenTelemetryPlan/Phase8_taskList.md b/OpenTelemetryPlan/Phase8_taskList.md index 8c4d80e80b..566041bd75 100644 --- a/OpenTelemetryPlan/Phase8_taskList.md +++ b/OpenTelemetryPlan/Phase8_taskList.md @@ -19,7 +19,7 @@ ## Task 8.1: Inject trace_id into Logs::format() -**Objective**: Add OTel trace context to every log line that is emitted within an active span. +**Objective**: Add OTel trace context to every log line that is emitted within an active, sampled span. The sampled flag matters because a span dropped by the `ParentBasedSampler` still carries its parent's ids, so emitting them would advertise a trace that was never exported. **What to do**: @@ -36,7 +36,7 @@ auto span = opentelemetry::nostd::get< opentelemetry::nostd::shared_ptr>(spanValue); auto spanCtx = span->GetContext(); - if (spanCtx.IsValid()) + if (spanCtx.IsValid() && spanCtx.IsSampled()) { char traceId[32], spanId[16]; spanCtx.trace_id().ToLowerBase16( @@ -62,7 +62,7 @@ - `src/libxrpl/basics/Log.cpp` -**Performance note**: The implementation checks the thread-local context value directly (avoiding the heap allocation that `GetSpan()` performs on the no-span path). On threads without an active span (~99% of log lines), the cost is a thread-local read + variant type check (~15-20ns). On the active-span path, an additional shared_ptr copy + `GetContext()` + `IsValid()` adds ~50ns total. Overhead is negligible at typical logging rates. +**Performance note**: The implementation checks the thread-local context value directly (avoiding the heap allocation that `GetSpan()` performs on the no-span path). On threads without an active span (~99% of log lines), the cost is a thread-local read + variant type check (~15-20ns). On the active-span path, an additional shared_ptr copy + `GetContext()` + `IsValid()`/`IsSampled()` adds ~50ns total. Overhead is negligible at typical logging rates. --- @@ -230,7 +230,7 @@ **Exit Criteria** (from [06-implementation-phases.md §6.8.1](./06-implementation-phases.md)): -- [ ] Log lines within active spans contain `trace_id= span_id=` +- [ ] Log lines within active, sampled spans contain `trace_id= span_id=` - [ ] Log lines outside spans have no trace context (no empty fields) - [ ] Loki ingests xrpld logs via OTel Collector filelog receiver - [ ] Grafana Tempo -> Loki one-click correlation works diff --git a/OpenTelemetryPlan/Phase9_taskList.md b/OpenTelemetryPlan/Phase9_taskList.md index 5c67ec1097..32aa098774 100644 --- a/OpenTelemetryPlan/Phase9_taskList.md +++ b/OpenTelemetryPlan/Phase9_taskList.md @@ -1,6 +1,11 @@ + + + # Phase 9: Internal Metric Instrumentation Gap Fill — Task List -> **Status**: Future Enhancement +> **Status**: Complete for Tasks 9.1-9.13. Tasks 9.14-9.17 remain open by design +> (see each task for the blocker). > > **Goal**: Instrument xrpld to emit ~50+ metrics that exist in `get_counts`/`server_info`/TxQ/PerfLog but currently lack time-series export via the OTel or beast::insight pipelines. > @@ -10,6 +15,36 @@ > > **Depends on**: Phase 7 (native OTel metrics pipeline) and Phase 8 (log-trace correlation) +> **Note on metric names**: there is **no `xrpld_` prefix** on any emitted +> metric. `77f35c03db` removed it and lowercased names, and +> `OTelCollectorImp::formatName()` +> (`src/libxrpl/beast/insight/OTelCollector.cpp:855-874`) adds no prefix at all — +> it only lowercases the raw name and turns `.` and spaces into `_`. Earlier +> revisions of this task list spelled every metric `xrpld_`; those spellings +> have been corrected in place to the emitted names, so the names below can be +> pasted into Prometheus as written. Instruments created in +> `src/xrpld/telemetry/MetricsRegistry.cpp` (35 of them) are the single source of +> truth. `MetricsRegistry.h`'s Doxygen used to disagree on three histogram names; +> those header comments were repaired in this change set (see Tasks 9.4 and 9.5), +> so header and `.cpp` now agree. +> +> **Two shapes do not simply lose the prefix**, so `xrpld_` → `` is +> not a blanket rule: +> +> - **Multiplexed observable gauges.** Most of the value names in these task +> descriptions are a **`metric` label value** on a shared instrument, not a +> standalone metric name — queue depth is `txq_metrics{metric="txq_count"}`, not +> `txq_count`. The same applies to `nodestore_state`, `cache_metrics`, +> `load_factor_metrics`, `server_info`, `db_metrics`, `validator_health`, +> `peer_quality`, `state_tracking` and `ledger_economy`. Each task below names +> its owning instrument. +> - **Unit-suffixed histograms** coming through `beast::insight`. +> `OTelCollectorImp` appends the unit to the name, so the `ios_latency` +> histogram is `ios_latency_milliseconds_bucket` in Prometheus — not +> `ios_latency_bucket`. Instruments created directly on `MetricsRegistry` keep +> their literal name (`job_queued_us_bucket`, `rpc_method_us_bucket`) because +> the unit is already in the instrument name. + ### Related Plan Documents | Document | Relevance | @@ -40,7 +75,16 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- In `src/libxrpl/nodestore/Database.cpp`, extend existing `beast::insight` registrations to add: +> **As shipped, this did _not_ go through `beast::insight`.** `Database.cpp` has +> no insight members. The metrics are a single `nodestore_state` +> `Int64ObservableGauge` on `MetricsRegistry` +> (`src/xrpld/telemetry/MetricsRegistry.cpp:957-965`) whose callback reads +> `Database`'s public accessors (`getFetchTotalCount()`, `getFetchHitCount()`, +> `getStoreCount()`, `getFetchDurationUs()`, `getStoreDurationUs()`, …) and +> multiplexes every value onto the `metric` label. Write-queue depth comes from +> the new `include/xrpl/nodestore/WriteStats.h`. + +- Export the following as `nodestore_state{metric="…"}` label values: - Gauge: `node_reads_total` (cumulative read operations) - Gauge: `node_reads_hit` (fetches that found an object — not a cache hit; `fetchHitCount_` increments whatever served the fetch) - Gauge: `node_writes` (cumulative write operations) @@ -50,14 +94,18 @@ These metrics serve multiple external consumer categories identified during rese - Gauge: `write_load` (current write load score) - Gauge: `read_queue` (items in read queue) -- These values are already computed in `Database::getCountsJson()` (line ~236). Wire the same counters to `beast::insight` hooks. +- These values are already computed in `Database::getCountsJson()`. The gauge + callback reads the same counters through `Database`'s public accessors. **Key modified files**: -- `src/libxrpl/nodestore/Database.cpp` -- `src/libxrpl/nodestore/Database.h` (add insight members) +- `src/xrpld/telemetry/MetricsRegistry.cpp` (the `nodestore_state` gauge) +- `include/xrpl/nodestore/Database.h` (accessors; **not** `src/libxrpl/nodestore/Database.h`, which does not exist) +- `include/xrpl/nodestore/WriteStats.h` (new — write-queue depth snapshot) -**Derived Prometheus metrics**: `xrpld_nodestore_reads_total`, `xrpld_nodestore_reads_hit`, `xrpld_nodestore_write_load`, etc. +**Derived Prometheus metrics**: `nodestore_state{metric="node_reads_total"}`, +`nodestore_state{metric="node_reads_hit"}`, `nodestore_state{metric="write_load"}`, +etc. There is **no** `xrpld_` prefix — `OTelCollectorImp::formatName()` adds none. **Grafana dashboard**: Add "NodeStore I/O" panel group to _Node Health_ dashboard. @@ -77,17 +125,22 @@ These metrics serve multiple external consumer categories identified during rese - `treenode_track_size` — Tracked tree nodes - `fullbelow_size` — FullBelow cache size -- The callback should read from the same sources as `GetCounts.cpp` handler (line ~43). +- The callback reads from the same sources as the `GetCounts` handler + (`src/xrpld/rpc/handlers/admin/status/GetCounts.cpp` — **not** + `src/xrpld/rpc/handlers/GetCounts.cpp`). - Create a centralized `MetricsRegistry` class that holds all OTel async gauge registrations, polled at 10-second intervals by the `PeriodicMetricReader`. **Key modified files**: - New: `src/xrpld/telemetry/MetricsRegistry.h` / `.cpp` -- `src/xrpld/rpc/handlers/GetCounts.cpp` (extract shared access methods) +- New: `src/xrpld/telemetry/MetricMacros.h` (the `XRPL_METRIC_*` call-site macros) +- `src/xrpld/rpc/handlers/admin/status/GetCounts.cpp` (extract shared access methods) - `src/xrpld/app/main/Application.cpp` (register MetricsRegistry at startup) -**Derived Prometheus metrics**: `xrpld_cache_SLE_hit_rate`, `xrpld_cache_ledger_hit_rate`, `xrpld_cache_treenode_size`, etc. +**Derived Prometheus metrics**: `cache_metrics{metric="SLE_hit_rate"}`, +`cache_metrics{metric="ledger_hit_rate"}`, `cache_metrics{metric="treenode_cache_size"}`, +etc. Label values are **case-sensitive** (`SLE_hit_rate`, `AL_size`, `AL_hit_rate`). --- @@ -97,7 +150,8 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- Register OTel `ObservableGauge` callbacks for TxQ state (from `TxQ.h` line ~143): +- Register OTel `ObservableGauge` callbacks for TxQ state (from + `src/xrpld/app/misc/TxQ.h` — **not** `src/xrpld/app/tx/detail/TxQ.h`): - `txq_count` — Current transactions in queue - `txq_max_size` — Maximum queue capacity - `txq_in_ledger` — Transactions in current open ledger @@ -112,9 +166,12 @@ These metrics serve multiple external consumer categories identified during rese **Key modified files**: - `src/xrpld/telemetry/MetricsRegistry.cpp` (add TxQ callbacks) -- `src/xrpld/app/tx/detail/TxQ.h` (expose metrics accessor if needed) +- `src/xrpld/app/misc/TxQ.h` (expose metrics accessor if needed) -**Derived Prometheus metrics**: `xrpld_txq_count`, `xrpld_txq_max_size`, `xrpld_txq_open_ledger_fee_level`, etc. +**Derived Prometheus metrics**: `txq_metrics{metric="txq_count"}`, +`txq_metrics{metric="txq_max_size"}`, `txq_metrics{metric="txq_open_ledger_fee_level"}`, etc. +There is one instrument, `txq_metrics` (`MetricsRegistry.cpp:705`); each value above +is a `metric` label value, not a metric name of its own. **Grafana dashboard**: New _Fee Market & TxQ_ dashboard (`fee-market`). @@ -126,13 +183,25 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp` line ~63): - - Counter: `xrpld_rpc_method_started_total{method=""}` — calls started - - Counter: `xrpld_rpc_method_finished_total{method=""}` — calls completed - - Counter: `xrpld_rpc_method_errored_total{method=""}` — calls errored - - Histogram: `xrpld_rpc_method_duration_us{method=""}` — execution time distribution +- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp`): + - Counter: `rpc_method_started_total{method=""}` — calls started + - Counter: `rpc_method_finished_total{method=""}` — calls completed + - Counter: `rpc_method_errored_total{method=""}` — calls errored + - Histogram: `rpc_method_us{method=""}` — execution time distribution -- Use OTel `Counter` and `Histogram` instruments with `method` attribute label. +- Use OTel `Counter` and `Histogram` instruments with the + `method` attribute label. The RPC instruments carry **only** `method` + (`MetricsRegistry.cpp:436-475`) — the `handler` label belongs to the job + instruments (Task 9.5), not these. + +> **Naming**: the instrument is `rpc_method_us` — declared as +> `kRpcMethodDurationUs` at `MetricsRegistry.cpp:96` and used both to register the +> explicit-bucket view and to create the instrument. `MetricsRegistry.h`'s Doxygen +> comment used to read `rpc_method_duration_us`; **that was fixed in this change** +> (`MetricsRegistry.h:789`), so header and `.cpp` now agree and there is no +> caveat left. The prefix `xrpld_` in the original spec is not emitted by anything. +> +> Same for the job histograms in Task 9.5: `job_queued_us` / `job_running_us`. - Hook into the existing PerfLog callback mechanism rather than adding new instrumentation points. @@ -141,7 +210,7 @@ These metrics serve multiple external consumer categories identified during rese - `src/xrpld/perflog/detail/PerfLogImp.cpp` (add OTel instrument updates alongside existing JSON counters) - `src/xrpld/telemetry/MetricsRegistry.cpp` (register instruments) -**Derived Prometheus metrics**: `xrpld_rpc_method_started_total{method="server_info"}`, `xrpld_rpc_method_duration_us_bucket{method="ledger"}`, etc. +**Derived Prometheus metrics**: `rpc_method_started_total{method="server_info"}`, `rpc_method_us_bucket{method="ledger"}`, etc. **Grafana dashboard**: Add "Per-Method RPC Breakdown" panel group to _RPC Performance_ dashboard. @@ -153,12 +222,24 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- Register OTel instruments for PerfLog job counters: - - Counter: `xrpld_job_queued_total{job_type=""}` — jobs queued - - Counter: `xrpld_job_started_total{job_type=""}` — jobs started - - Counter: `xrpld_job_finished_total{job_type=""}` — jobs completed - - Histogram: `xrpld_job_queued_duration_us{job_type=""}` — time spent waiting in queue - - Histogram: `xrpld_job_running_duration_us{job_type=""}` — execution time distribution +- Register OTel instruments for PerfLog job counters. All five carry **two** + labels — `job_type` and `handler` — so producers sharing a job type stay + distinguishable (`MetricsRegistry.h:794-818`, recorded at + `MetricsRegistry.cpp:498,518,527,548,553`). `handler` is the sanitised + `addJob` name; `sanitiseHandler()` folds dynamic names into a bounded domain + of exactly 44 values, so cardinality stays fixed. + - Counter: `job_queued_total{job_type="",handler=""}` — jobs queued + - Counter: `job_started_total{job_type="",handler=""}` — jobs started + - Counter: `job_finished_total{job_type="",handler=""}` — jobs completed + - Histogram: `job_queued_us{job_type="",handler=""}` — time spent waiting in queue + - Histogram: `job_running_us{job_type="",handler=""}` — execution time distribution + +> **Naming**: the instruments are `job_queued_us` / `job_running_us` +> (`kJobQueuedDurationUs` / `kJobRunningDurationUs`, `MetricsRegistry.cpp:94-95`). +> `MetricsRegistry.h`'s Doxygen comments used to read +> `job_queued_duration_us` / `job_running_duration_us`; **both were fixed in this +> change** (`MetricsRegistry.h:810,815`), so there is no header/`.cpp` divergence +> left to work around. - Hook into PerfLog's existing job tracking alongside Task 9.4. @@ -167,7 +248,7 @@ These metrics serve multiple external consumer categories identified during rese - `src/xrpld/perflog/detail/PerfLogImp.cpp` - `src/xrpld/telemetry/MetricsRegistry.cpp` -**Derived Prometheus metrics**: `xrpld_job_queued_total{job_type="ledgerData"}`, `xrpld_job_running_duration_us_bucket{job_type="transaction"}`, etc. +**Derived Prometheus metrics**: `job_queued_total{job_type="ledgerData",handler="ProcessLData"}`, `job_running_us_bucket{job_type="transaction",handler="…"}`, etc. **Grafana dashboard**: New _Job Queue Analysis_ dashboard (`job-queue`). @@ -180,15 +261,16 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: - Register OTel `ObservableGauge` callbacks for `CountedObject` instance counts: - - `xrpld_object_count{type="Transaction"}` — live Transaction objects - - `xrpld_object_count{type="Ledger"}` — live Ledger objects - - `xrpld_object_count{type="NodeObject"}` — live NodeObject instances - - `xrpld_object_count{type="STTx"}` — serialized transaction objects - - `xrpld_object_count{type="STLedgerEntry"}` — serialized ledger entries - - `xrpld_object_count{type="InboundLedger"}` — ledgers being fetched - - `xrpld_object_count{type="Pathfinder"}` — active pathfinding computations - - `xrpld_object_count{type="PathRequest"}` — active path requests - - `xrpld_object_count{type="HashRouterEntry"}` — hash router entries + - `object_count{type="xrpl::Transaction"}` — live Transaction objects + - `object_count{type="xrpl::Ledger"}` — live Ledger objects + - `object_count{type="xrpl::NodeObject"}` — live NodeObject instances + - `object_count{type="xrpl::STTx"}` — serialized transaction objects + - `object_count{type="xrpl::STLedgerEntry"}` — serialized ledger entries + - `object_count{type="xrpl::InboundLedger"}` — ledgers being fetched + - `object_count{type="xrpl::Pathfinder"}` — active pathfinding computations + - `object_count{type="xrpl::PathRequest"}` — active path requests + - `object_count{type="xrpl::HashRouter::Entry"}` — hash router entries (the type is + `HashRouter::Entry`; there is no `HashRouterEntry` type) - The `CountedObject` template already tracks these via atomic counters. The callback just reads the current counts. @@ -197,7 +279,9 @@ These metrics serve multiple external consumer categories identified during rese - `src/xrpld/telemetry/MetricsRegistry.cpp` (add counted object callbacks) - `include/xrpl/basics/CountedObject.h` (may need static accessor for iteration) -**Derived Prometheus metrics**: `xrpld_object_count{type="Transaction"}`, `xrpld_object_count{type="NodeObject"}`, etc. +**Derived Prometheus metrics**: `object_count{type="xrpl::Transaction"}`, `object_count{type="xrpl::NodeObject"}`, etc. +The `type` label value is `beast::typeName()` — the fully-qualified +demangled C++ type name (`CountedObject.h:109`), not a short word. **Grafana dashboard**: Add "Object Instance Counts" panel to _Node Health_ dashboard. @@ -225,7 +309,10 @@ These metrics serve multiple external consumer categories identified during rese - `src/xrpld/telemetry/MetricsRegistry.cpp` - `src/xrpld/app/misc/NetworkOPs.cpp` (expose load factor accessors if needed) -**Derived Prometheus metrics**: `xrpld_load_factor`, `xrpld_load_factor_fee_escalation`, etc. +**Derived Prometheus metrics**: `load_factor_metrics{metric="load_factor"}`, +`load_factor_metrics{metric="load_factor_fee_escalation"}`, etc. There is one +instrument, `load_factor_metrics` (`MetricsRegistry.cpp:785`); every value listed +above is a `metric` label value, not a metric name of its own. **Grafana dashboard**: Add "Load Factor Breakdown" panel to _Fee Market & TxQ_ dashboard. @@ -243,7 +330,7 @@ These metrics serve multiple external consumer categories identified during rese - `read_request_bundle` (native JSON int) - `read_threads_running` (native JSON int) - `read_threads_total` (native JSON int) -- Added new `xrpld_server_info` Int64ObservableGauge with 8 metrics: +- Added new `server_info` Int64ObservableGauge with 8 metrics: - `server_state` — operating mode as int (0=DISCONNECTED .. 4=FULL) - `uptime` — seconds since server start - `peers` — total peer count @@ -252,9 +339,9 @@ These metrics serve multiple external consumer categories identified during rese - `peer_disconnects_resources` — cumulative resource-related disconnects - `last_close_proposers` — from `getConsensusInfo()["previous_proposers"]` - `last_close_converge_time_ms` — from `getConsensusInfo()["previous_mseconds"]` -- Added new `xrpld_build_info` Int64ObservableGauge (info-style, value=1 with `version` label) -- Added new `xrpld_complete_ledgers` Int64ObservableGauge parsing comma-separated ranges into `{bound, index}` pairs -- Added new `xrpld_db_metrics` Int64ObservableGauge with 4 metrics: +- Added new `build_info` Int64ObservableGauge (info-style, value=1 with `version` label) +- Added new `complete_ledgers` Int64ObservableGauge parsing comma-separated ranges into `{bound, index}` pairs +- Added new `db_metrics` Int64ObservableGauge with 4 metrics: - `db_kb_total`, `db_kb_ledger`, `db_kb_transaction` (SQLite stat queries) - `historical_perminute` (historical ledger fetch rate) @@ -267,7 +354,7 @@ These metrics serve multiple external consumer categories identified during rese - `connection_count_51233/51234` — OS-level port connection counts from external shell script (`get_connection.sh`) -**Derived Prometheus metrics**: `xrpld_server_info{metric="server_state"}`, `xrpld_build_info{version="2.4.0"}`, `xrpld_complete_ledgers{bound="start",index="0"}`, `xrpld_db_metrics{metric="db_kb_total"}`, etc. +**Derived Prometheus metrics**: `server_info{metric="server_state"}`, `build_info{version="2.4.0"}`, `complete_ledgers{bound="start",index="0"}`, `db_metrics{metric="db_kb_total"}`, etc. **Grafana dashboard**: New panels added to _Node Health_ dashboard (`node-health.json`). @@ -284,15 +371,20 @@ These metrics serve multiple external consumer categories identified during rese 2. **Job Queue Analysis** (`job-queue`) — Per-job-type rates, queue wait times, execution times, job queue depth - Update 2 existing dashboards: - 1. **Node Health** (`xrpld-statsd-node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts + 1. **Node Health** (`node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts 2. **RPC Performance** (`rpc-performance`) — Add per-method RPC breakdown panels -**Key modified files**: +> Tasks 9.11-9.13 add two more new dashboards (`validator-health`, +> `peer-quality`), so Phase 9's total is **4 new + 2 updated**. -- New: `docker/telemetry/grafana/dashboards/rippled-fee-market.json` -- New: `docker/telemetry/grafana/dashboards/rippled-job-queue.json` -- `docker/telemetry/grafana/dashboards/rippled-statsd-node-health.json` -- `docker/telemetry/grafana/dashboards/rippled-rpc-perf.json` +**Key modified files** (filenames and uids after the `dashboards/rippled-*` → +bare rename in `145b1469d6` and `25868f2740` — the +`dashboards/rippled-*.json` paths no longer exist): + +- New: `docker/telemetry/grafana/dashboards/fee-market.json` (uid `fee-market`) +- New: `docker/telemetry/grafana/dashboards/job-queue.json` (uid `job-queue`) +- `docker/telemetry/grafana/dashboards/node-health.json` (uid `node-health`) +- `docker/telemetry/grafana/dashboards/rpc-performance.json` (uid `rpc-performance`) --- @@ -302,18 +394,37 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- Update `OpenTelemetryPlan/09-data-collection-reference.md`: - - Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors) - - Update Grafana dashboard reference table (add 2 new dashboards) +- Update `OpenTelemetryPlan/09-data-collection-reference.md`: ✅ done + - Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors) — §5b + "Phase 9: OTel SDK-Exported Metrics (MetricsRegistry)" + - Update Grafana dashboard reference table (add 4 new dashboards) — "New Grafana Dashboards (Phase 9)" / "Updated Grafana Dashboards (Phase 9)" - Add Prometheus query examples for new metrics - Update `docs/telemetry-runbook.md`: - - Add an Alerting section covering the provisioned rules and how to wire a receiver - - Add troubleshooting entries for new metric categories + - ✅ Alerting section covering the provisioned rules and how to wire a receiver + - ✅ Troubleshooting entries for new metric categories + - ❌ **Still open**: dashboard guides for **six** dashboards — `fee-market`, + `job-queue`, `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality` and + `validator-health`. The runbook's dashboard reference records the gap + verbatim: "Nine dashboards have a reference section below. `fee-market`, + `job-queue`, `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality`, and + `validator-health` are provisioned but not yet documented here — their panel + descriptions carry the same six-heading reference format, so open the panel + info icon in Grafana until a section is written." (15 provisioned − 6 + undocumented = 9 documented.) Also still open: the Validation Agreement + explainer (8s grace / 5m late repair) -- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`): - - 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`) - - `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change) +- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`) — **as shipped**: + - **13 rules in 5 groups**: `xrpld-consensus` (`LedgerHistoryMismatch`, + `LedgerCloseStalled`, `ValidatedLedgerStale`), `xrpld-validator` + (`ValidationsMissed`, `ValidationsNotChecked`), `xrpld-jobqueue` + (`JobQueueTxOverflow`, `JobQueueLatencyHigh`, `NodeStoreIOLatencyHigh`), + `xrpld-node-state` (`NodeStateFlapping`, `NodeNotFull`), `xrpld-overlay` + (`ManifestJobQueueConvoy`, `ManifestFloodInbound`, `PeerResourceDisconnects`) + - **2 contact points** — `xrpld-default` (Slack) and `xrpld-critical` + (Slack + email) — and a **nested** notification policy: root → + `xrpld-default`, child route `severity = critical` → `xrpld-critical`. + Auto-loaded via the existing `provisioning/` mount (no docker-compose change) + - 3 rules are `severity: critical`, 10 are `severity: warning` - Alerting operator docs (per-alert meaning, tuning, receiver wiring) now live in the Alerting section of `docs/telemetry-runbook.md` **Key modified files**: @@ -331,21 +442,35 @@ These metrics serve multiple external consumer categories identified during rese **What to do**: -- Extend the existing telemetry integration test: - - Start xrpld with `[telemetry] enabled=1` and `[insight] server=otel` - - Submit a batch of RPC calls and transactions - - Query Prometheus for each new metric family - - Assert non-zero values for: NodeStore reads, cache hit rates, TxQ count, PerfLog RPC counters, object counts, load factors +- ❌ **Not done on this branch**: extend the telemetry integration test to + start xrpld with `[telemetry] enabled=1` / `[insight] server=otel`, drive RPC + and transaction load, query Prometheus for each new metric family and assert + non-zero values. The end-to-end metric assertions live in the **Phase 10** + harness (`docker/telemetry/workload/expected_metrics.json`), not here. -- Add unit tests for the `MetricsRegistry` class: - - Verify callback registration and deregistration - - Verify metric values match `get_counts` JSON output - - Verify graceful behavior when telemetry is disabled +- ✅ **Done**: unit tests for the `MetricsRegistry` class — + `src/tests/libxrpl/telemetry/MetricsRegistry.cpp` (**18** GTest cases — + `grep -cE '\bTEST(_F|_P)?\s*\(' src/tests/libxrpl/telemetry/MetricsRegistry.cpp` + = 18, and the four bullets below sum to 4 + 3 + 5 + 6 = 18): + - Callback registration / deregistration and shutdown ordering — + `async_gauges_start_after_start_is_safe`, + `async_gauges_before_start_does_not_break_start`, + `async_gauges_respect_the_compile_time_guard`, `destructor_calls_stop` + - Graceful behaviour when telemetry is disabled — `disabled_construction`, + `disabled_start_stop`, `disabled_recording_methods` + - Label sanitisation and mean scaling — `MetricsRegistrySanitiseHandler` (5 + cases, incl. `output_domain_is_exactly_44_values`) and + `MetricsRegistryScaledMean` (6 cases) + - ❌ Not covered: asserting metric values match `get_counts` JSON output — + that needs a live `Application`, so it is left to the Phase 10 harness -**Key modified files**: +**Key files**: -- `src/test/telemetry/MetricsRegistry_test.cpp` (new) -- Existing integration test script (extend assertions) +- `src/tests/libxrpl/telemetry/MetricsRegistry.cpp` (new). The originally + planned `src/test/telemetry/MetricsRegistry_test.cpp` was **never created** — + Phase 9 tests are GTest under `src/tests/libxrpl/`, per project convention. +- `src/tests/libxrpl/telemetry/MetricMacros.cpp`, `GetMeter.cpp` (new — cover + the `XRPL_METRIC_*` macros and meter lookup) --- @@ -360,31 +485,44 @@ These metrics serve multiple external consumer categories identified during rese **Dashboard**: `validator-health.json` -| Panel | Type | PromQL | -| -------------------------- | ---------- | -------------------------------------------------------------- | -| Agreement % (1h) | stat | `xrpld_validation_agreement{metric="agreement_pct_1h"}` | -| Agreement % (24h) | stat | `xrpld_validation_agreement{metric="agreement_pct_24h"}` | -| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side | -| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side | -| Validation Rate | stat | `rate(xrpld_validations_sent_total[5m]) * 60` | -| Validations Checked Rate | stat | `rate(xrpld_validations_checked_total[5m]) * 60` | -| Amendment Blocked | stat | `xrpld_validator_health{metric="amendment_blocked"}` | -| UNL Expiry (days) | stat | `xrpld_validator_health{metric="unl_expiry_days"}` | -| Validation Quorum | stat | `xrpld_validator_health{metric="validation_quorum"}` | -| State Value Timeline | timeseries | `xrpld_state_tracking{metric="state_value"}` | -| Time in Current State | stat | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` | -| State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` | -| Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` | +| Panel | Type | PromQL | +| -------------------------- | ---------- | -------------------------------------------------------- | +| Agreement % (1h) | stat | `validation_agreement{metric="agreement_pct_1h"}` | +| Agreement % (24h) | stat | `validation_agreement{metric="agreement_pct_24h"}` | +| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side | +| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side | +| Validation Rate | stat | `rate(validations_sent_total[5m]) * 60` | +| Validations Checked Rate | stat | `rate(validations_checked_total[5m]) * 60` | +| Amendment Blocked | stat | `validator_health{metric="amendment_blocked"}` | +| UNL Expiry (days) | stat | `validator_health{metric="unl_expiry_days"}` | +| Validation Quorum | stat | `validator_health{metric="validation_quorum"}` | +| State Value Timeline | timeseries | `state_tracking{metric="state_value"}` | +| Time in Current State | stat | `state_tracking{metric="time_in_current_state_seconds"}` | +| State Changes Rate | stat | `rate(state_changes_total[1h])` | +| Ledgers Closed Rate | stat | `rate(ledgers_closed_total[5m]) * 60` | **Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes. -**Key new files**: `docker/telemetry/grafana/dashboards/rippled-validator-health.json` +**Key new files**: `docker/telemetry/grafana/dashboards/validator-health.json` +(uid `validator-health`). The name reached its current form in **two** renames: +`dashboards/rippled-validator-health.json` → `xrpld-validator-health.json` +(`145b1469d6`, the `dashboards/rippled-*` → `xrpld-*` pass), then +`xrpld-validator-health.json` → `validator-health.json` (`25868f2740`, which +dropped the `xrpld-` prefix). **Exit Criteria**: -- [ ] All 13 panels render with non-zero data during normal operation -- [ ] `$node` filter works correctly for multi-node deployments -- [ ] Amendment blocked and UNL expiry panels use color thresholds (red=blocked/expiring) +- [x] Dashboard ships **17** panels (4 more than the 13 planned above) across 3 + rows — Validation Agreement, Validation Rates, Server State & Consensus +- [ ] All panels render with non-zero data during normal operation — needs a live + stack; the Phase 10 harness asserts the dashboard _loads_, not that panels + are non-empty +- [x] `$node` filter works correctly for multi-node deployments — `node` + template variable present (filters on `service_instance_id`), alongside + `service_name`, `deployment_environment`, `xrpl_network_type`, + `xrpl_work_item`, `xrpl_branch`, `xrpl_node_role` +- [x] Amendment blocked and UNL expiry panels use color thresholds + (red=blocked/expiring) — 11 `thresholds` blocks in the dashboard JSON --- @@ -396,22 +534,36 @@ These metrics serve multiple external consumer categories identified during rese **Dashboard**: `peer-quality.json` -| Panel | Type | PromQL | -| ---------------------- | ---------- | -------------------------------------------------------------- | -| P90 Peer Latency | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}` | -| Insane/Diverged Peers | stat | `xrpld_peer_quality{metric="peers_insane_count"}` | -| Higher Version Peers % | stat | `xrpld_peer_quality{metric="peers_higher_version_pct"}` | -| Upgrade Recommended | stat | `xrpld_peer_quality{metric="upgrade_recommended"}` | -| Resource Disconnects | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges` | -| Inbound vs Outbound | bargauge | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` | +| Panel | Type | PromQL | +| ---------------------- | ---------- | ----------------------------------------------------------------------- | +| P90 Peer Latency | timeseries | `peer_quality{metric="peer_latency_p90_ms"}` | +| Insane/Diverged Peers | stat | `peer_quality{metric="peers_insane_count"}` | +| Higher Version Peers % | stat | `peer_quality{metric="peers_higher_version_pct"}` | +| Upgrade Recommended | stat | `peer_quality{metric="upgrade_recommended"}` | +| Resource Disconnects | timeseries | `server_info{metric="peer_disconnects_resources"}` | +| Inbound vs Outbound | bargauge | `peer_finder_active_inbound_peers`, `peer_finder_active_outbound_peers` | -**Key new files**: `docker/telemetry/grafana/dashboards/rippled-peer-quality.json` +> `overlay_peer_disconnects_charges` (the name in the original spec) is **not a +> real instrument** — nothing registers it. The shipped panel reads +> `server_info{metric="peer_disconnects_resources"}` instead. Peer-finder gauge +> names are lowercase: `GroupImp::makeName()` + `OTelCollectorImp::formatName()` +> turn the `"Peer_Finder"` group into `peer_finder_` with no prefix. + +**Key new files**: `docker/telemetry/grafana/dashboards/peer-quality.json` +(uid `peer-quality`). Two renames, same as Task 9.11: +`dashboards/rippled-peer-quality.json` → `xrpld-peer-quality.json` +(`145b1469d6`), then `xrpld-peer-quality.json` → `peer-quality.json` +(`25868f2740`). **Exit Criteria**: -- [ ] All 6 panels render correctly -- [ ] P90 latency panel shows trend over time -- [ ] Upgrade recommended panel uses color threshold (red=1, green=0) +- [x] All 6 panels present — P90 Peer Latency, Insane/Diverged Peers, Higher + Version Peers %, Upgrade Recommended, Inbound vs Outbound Peers, Resource + Disconnects — across 3 rows, with the `$node` template variable +- [ ] All 6 panels render with data — needs a live stack +- [x] P90 latency panel is a `timeseries` (shows trend over time) +- [x] Upgrade recommended panel uses color threshold (red=1, green=0) — 5 + `thresholds` blocks in the dashboard JSON --- @@ -421,21 +573,22 @@ These metrics serve multiple external consumer categories identified during rese **Objective**: Add "Ledger Economy" row to the existing `node-health.json` dashboard. -| Panel | Type | PromQL | -| -------------------- | ---------- | --------------------------------------------------- | -| Base Fee (drops) | stat | `xrpld_ledger_economy{metric="base_fee_xrp"}` | -| Reserve Base (drops) | stat | `xrpld_ledger_economy{metric="reserve_base_xrp"}` | -| Reserve Inc (drops) | stat | `xrpld_ledger_economy{metric="reserve_inc_xrp"}` | -| Ledger Age | stat | `xrpld_ledger_economy{metric="ledger_age_seconds"}` | -| Transaction Rate | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}` | +| Panel | Type | PromQL | +| -------------------- | ---------- | --------------------------------------------- | +| Base Fee (drops) | stat | `ledger_economy{metric="base_fee_xrp"}` | +| Reserve Base (drops) | stat | `ledger_economy{metric="reserve_base_xrp"}` | +| Reserve Inc (drops) | stat | `ledger_economy{metric="reserve_inc_xrp"}` | +| Ledger Age | stat | `ledger_economy{metric="ledger_age_seconds"}` | +| Transaction Rate | timeseries | `ledger_economy{metric="transaction_rate"}` | **Key modified files**: `docker/telemetry/grafana/dashboards/node-health.json` **Exit Criteria**: -- [ ] 5 new panels render correctly in existing dashboard -- [ ] Fee values match `server_info` RPC output -- [ ] Transaction rate shows smooth trend (not spiky) +- [x] 5 new panels present in the existing dashboard — a "Ledger Economy" row + with 5 `ledger_economy` queries is on `node-health.json` +- [ ] Fee values match `server_info` RPC output — needs a live comparison +- [ ] Transaction rate shows smooth trend (not spiky) — needs a live run --- @@ -456,9 +609,22 @@ files, so **no code fix lands on this branch**. **Why deferred**: Defect 3 requires widening the two `OverlayImpl::updateSlotAndSquelch` overloads — a public signature change on -shared overlay code. Defects 1, 2 and 4 sit in `TrafficCount.{h,cpp}`, likewise -not telemetry-owned. Routing them through the telemetry chain would hide overlay -changes from overlay reviewers and couple them to a 12-PR merge timeline. +shared overlay code. Defects 1 and 4 need `TrafficCount.cpp` and `PeerImp.cpp` +edits that are not telemetry-owned. Routing them through the telemetry chain +would hide overlay changes from overlay reviewers and couple them to a 12-PR +merge timeline. + +> **Constraint narrowed.** The blanket "no telemetry change may touch +> `TrafficCount.{h,cpp}`" no longer holds for the header: the telemetry chain +> already edits `TrafficCount.h` — Phase 6's `77f35c03db` fixed the +> `Category::GetFetchPack` label from `"getobject_Fetch Pack_get"` to +> `"getobject_Fetch_Pack_get"` at `TrafficCount.h:285`, the sole difference from +> `develop`. Defect 2 (the stale `Total` header comment, `TrafficCount.h:28-31`) +> is therefore **unblocked** and can land here. Defects **1, 3 and 4** stay +> blocked: defect 1 needs `TrafficCount.cpp`'s `kTypeLookup`, defect 3 needs the +> `OverlayImpl` signature change, and defect 4 needs `PeerImp.cpp:1079` vs `:313` +> to agree on a byte basis (compressed vs uncompressed) — a change to overlay +> accounting semantics, not telemetry. **Key modified files**: `OpenTelemetryPlan/09-data-collection-reference.md` only. @@ -466,16 +632,31 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline. - [x] Each defect documented with file:line evidence in `09` §6 - [x] `overhead_cluster_*` documented as "no data", not "no cluster traffic" -- [ ] Follow-up overlay-owned branch raised for the four code fixes +- [ ] Defect 2 (stale `Total` header comment, `TrafficCount.h:28-31`) fixed on + this branch — it is **unblocked** (the chain already edits + `TrafficCount.h`) but the comment is still uncorrected +- [ ] Follow-up overlay-owned branch raised for the three still-blocked code + fixes (defects 1, 3, 4) - [ ] Re-baseline any threshold keyed on `unknown_bytes_in` when defect 1 lands --- ## Task 9.15: Peer Keepalive and Discovery Instrumentation -> **Status**: NOT IMPLEMENTED — awaiting a decision on whether `XRPL_METRIC_*` -> call sites may be added to `src/xrpld/overlay/detail/PeerImp.cpp` from this -> branch. Reference: [09 §6.3](./09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) +> **Status**: NOT IMPLEMENTED. The instruments themselves are still to be +> written; the _permission_ question is settled. Reference: +> [09 §6.3](./09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) +> +> **Blocker cleared.** This task used to be held "awaiting a decision on whether +> `XRPL_METRIC_*` call sites may be added to +> `src/xrpld/overlay/detail/PeerImp.cpp` from this branch". That decision is +> de facto **yes** — `PeerImp.cpp` already carries **7** such call sites on this +> branch (`:2723`, `:2741`, `:2925`, `:2928`, `:2931`, `:2947`, `:2954`, of which +> three are `XRPL_METRIC_HISTOGRAM_RECORD` — `:2925`, `:2928`, `:2931` — and four +> are labelled counters — `:2723`, `:2741`, `:2947`, `:2954`). Note that +> `grep -c XRPL_METRIC src/xrpld/overlay/detail/PeerImp.cpp` returns 8: the eighth +> hit is the `cspell:ignore` explanation comment at `PeerImp.cpp:2`, not a call +> site. What remains is the implementation work below, not an approval. **Objective**: Make peer keepalive and peer-discovery health observable. Today `mtPING`, `mtSTATUS_CHANGE` and `mtENDPOINTS` are byte counters only. @@ -494,7 +675,8 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline. - `peer_id` as a label is unbounded cardinality — rejected. A bounded `peer_role`-style label is the alternative if per-peer attribution is needed. - Splitting `mtPING` out of `Category::Base` is a `TrafficCount.cpp` change and - therefore blocked with Task 9.14. + therefore still blocked with Task 9.14 defect 1. (The `.h` half of that + constraint no longer applies — see Task 9.14.) - Per the runbook's "Adding a New Metric" contract, `_total` is reserved for monotonic counters; a histogram takes no suffix. @@ -505,7 +687,8 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline. **Exit Criteria**: -- [ ] Decision recorded on editing `PeerImp.cpp` from the telemetry chain +- [x] Decision recorded on editing `PeerImp.cpp` from the telemetry chain — yes; + 7 `XRPL_METRIC_*` call sites already ship in `PeerImp.cpp` - [ ] Three instruments emitting, with an explicit histogram bucket view - [ ] Rows added to `09` §5b, runbook § Metric Reference, and `expected_metrics.json` - [ ] Peer Quality dashboard panels follow the Task 9.12 conventions (`$node`, Title Case, legend dimensions) @@ -557,10 +740,11 @@ actually emits. `peer.connect`, `peer.disconnect`, `peer.message.send` and protocol message families have no spans. **Scope warning**: This is larger than Tasks 9.14-9.16 combined and changes the -"~37 spans" figure asserted in `09` §1.1 and in -`docker/telemetry/workload/expected_spans.json`. `trace_peer` is also **on by -default** and already flagged as high-volume, so adding per-message spans has a -volume cost that needs measuring before commitment. +span-family inventory asserted in `09` §1.1 (**41** emitted families) and in +`docker/telemetry/workload/expected_spans.json` (**40** catalogued — `rpc.ws_upgrade` +has no entry). `trace_peer` is also **on by default** and already flagged as +high-volume, so adding per-message spans has a volume cost that needs measuring +before commitment. **Exit Criteria**: @@ -572,17 +756,41 @@ volume cost that needs measuring before commitment. ## Exit Criteria -- [ ] All ~50 new metrics visible in Prometheus via OTLP pipeline -- [ ] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK -- [ ] Async gauge callbacks execute at 10s intervals without performance impact -- [ ] 2 new Grafana dashboards operational (Fee Market, Job Queue) -- [ ] 2 existing dashboards updated with new panel groups -- [ ] Integration test validates all new metric families are non-zero -- [ ] No performance regression (< 0.5% CPU overhead from new callbacks) -- [ ] Documentation updated with full new metric inventory -- [ ] Validator Health dashboard renders all 13 panels -- [ ] Peer Quality dashboard renders all 6 panels -- [ ] Ledger Economy panels added to node-health dashboard +- [ ] All ~50 new metrics visible in Prometheus via OTLP pipeline — every + instrument is registered in `MetricsRegistry.cpp`, but end-to-end + visibility is asserted only by the Phase 10 harness +- [x] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK — + `src/tests/libxrpl/telemetry/MetricsRegistry.cpp` + (`async_gauges_start_after_start_is_safe`, + `async_gauges_before_start_does_not_break_start`, + `async_gauges_respect_the_compile_time_guard`, `destructor_calls_stop`) +- [x] Async gauge callbacks execute at 10s intervals — + `MetricsRegistry.cpp:289`, `readerOpts.export_interval_millis = 10000`. + (The "without performance impact" half is unmeasured — see below.) +- [x] 4 new Grafana dashboards operational (Fee Market, Job Queue, Validator + Health, Peer Quality) — all four JSONs are under + `docker/telemetry/grafana/dashboards/` +- [x] 2 existing dashboards updated with new panel groups — `node-health` + (NodeStore I/O, Caches, Server Info, Complete Ledgers & DB, Ledger + Economy, Job Queue Concurrency Limits rows) and `rpc-performance` + (per-method section) +- [ ] Integration test validates all new metric families are non-zero — not on + this branch; lives in the Phase 10 harness (`expected_metrics.json`) +- [ ] No performance regression (< 0.5% CPU overhead from new callbacks) — not + measured; needs the Phase 10 benchmark suite +- [x] Documentation updated with full new metric inventory — + `09-data-collection-reference.md` §5b + "Phase 9: OTel SDK-Exported + Metrics (MetricsRegistry)" + "Phase 7+: External Dashboard Parity Metrics" +- [x] Validator Health dashboard ships (17 panels, 4 more than the 13 planned) +- [x] Peer Quality dashboard ships (6 panels) +- [x] Ledger Economy panels added to node-health dashboard (5 panels in a + "Ledger Economy" row) +- [x] Provisioned Grafana alerting: 13 rules / 5 groups, 2 contact points, + nested notification policy +- [ ] Tasks 9.14-9.17 closed — **open by design**: 9.14 documented-not-fixed + (defects 1, 3 and 4 still blocked; defect 2 unblocked but not yet fixed), + 9.15 and 9.16 not implemented, 9.17 deferred pending approval and volume + measurement --- @@ -591,7 +799,7 @@ volume cost that needs measuring before commitment. > Design for the provisioned Grafana alert rules (Task 9.9a). Previously a standalone spec; merged here so the phase plan is self-contained. **Date:** 2026-07-06 -**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187) +**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513) **Status:** Approved ### Purpose @@ -599,8 +807,8 @@ volume cost that needs measuring before commitment. Phase 9 exports ~68 internal xrpld metrics and ships Grafana dashboards for them. This adds the missing operator-facing piece: **provisioned Grafana alert rules** that fire on the health-critical metrics phase 9 introduces. The -phase-9 task list (line 311) and Jira story RIPD-5187 both already list -"alerting rules" as a phase-9 deliverable, so this closes that gap. +phase-9 task list already lists "alerting rules" as a phase-9 deliverable +(Task 9.9), so this closes that gap. Scope is deliberately narrow — the three subsystems whose failure is node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API @@ -609,9 +817,11 @@ health is explicitly out of scope. ### Why phase 9 (not phase 11) Every metric these alerts fire on is _born_ in phase 9 -(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`, -`xrpld_validation_missed_total`, `xrpld_validations_checked_total`, -`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts +(`ledger_history_mismatch_total`, `ledgers_closed_total`, +`validation_missed_total`, `validations_checked_total`, +`jq_trans_overflow_total`, `job_queued_us_bucket` — the histogram instrument is +`job_queued_us` (`MetricsRegistry.cpp:94`), so the Prometheus bucket series is +`job_queued_us_bucket`, not `job_queued_duration_us_bucket`). Alerts belong with the metrics they watch, and this is where the dependency lives. ### Delivery @@ -623,11 +833,11 @@ Grafana auto-loads `provisioning/alerting/*.yaml`. New files under `docker/telemetry/grafana/provisioning/alerting/`: -| File | Purpose | -| -------------------- | ------------------------------------------------------------------------------------------------------------------- | -| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). | -| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `service_instance_id`. | -| `rules.yaml` | 6 alert rules across 3 groups (below). | +| File | Purpose | +| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `contactpoints.yaml` | **Two** contact points: `xrpld-default` (Slack) and `xrpld-critical` (Slack + email). | +| `policies.yaml` | **Nested** notification policy: root route → `xrpld-default`; child route matching `severity = critical` → `xrpld-critical` (`repeat_interval: 1h` vs the root's `4h`). Both grouped by `alertname` + `service_instance_id`. | +| `rules.yaml` | **13** alert rules across **5** groups (below). | Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook: what each alert means, likely causes, and how to point the contact point at a @@ -642,16 +852,44 @@ Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All Alert rules run headless, so they cannot use the dashboards' `$node` template variables — they match all series and group by `service_instance_id` instead. -| Group | Alert | Expression (5m window) | Fires | `for` | severity | -| --------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- | -| Consensus | LedgerHistoryMismatch | `sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical | -| Consensus | LedgerCloseStalled | `sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical | -| Validator | ValidationsMissed | `sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning | -| Validator | ValidationsNotChecked | `sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning | -| Job queue | JobQueueTxOverflow | `sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning | -| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning | +All 5 groups evaluate at `interval: 1m`. Metric names carry **no** `xrpld_` +prefix — `OTelCollectorImp::formatName()` adds none. -Each rule carries labels `severity` and `category` (consensus/validator/jobqueue) +The **Threshold** column is the rule's refId `C` evaluator, read straight from +`rules.yaml` — it is the firing condition, so it is load-bearing, not decoration. + +| Group | Alert | Expression (refId A) | Threshold (refId C) | `for` | severity | +| ------------------ | ----------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----- | -------- | +| `xrpld-consensus` | LedgerHistoryMismatch | `sum by (service_instance_id) (increase(ledger_history_mismatch_total[15m]))` | `gt [0]` | 2m | critical | +| `xrpld-consensus` | LedgerCloseStalled | `rate(ledgers_closed_total)` decayed to ≈0 | `lt [0.001]` | 3m | critical | +| `xrpld-consensus` | ValidatedLedgerStale | `max by (service_instance_id) (ledgermaster_validated_ledger_age < 1209600)` | `gt [60]` (seconds) | 5m | critical | +| `xrpld-validator` | ValidationsMissed | miss **ratio**, gated on send activity — see the expression below the table | `gt [0.1]` | 15m | warning | +| `xrpld-validator` | ValidationsNotChecked | `rate(validations_checked_total)` ≈0 | `lt [0.001]` | 5m | warning | +| `xrpld-jobqueue` | JobQueueTxOverflow | `sum by (service_instance_id) (increase(jq_trans_overflow_total[15m]))` | `gt [0]` | 2m | warning | +| `xrpld-jobqueue` | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket[5m])))` | `gt [1000000]` (µs = 1s) | 5m | warning | +| `xrpld-jobqueue` | NodeStoreIOLatencyHigh | `histogram_quantile(0.95, sum by (le, service_instance_id) (rate(ios_latency_milliseconds_bucket[10m])))` | `gt [1000]` (ms) | 10m | warning | +| `xrpld-node-state` | NodeStateFlapping | state-transition rate over the node-state series | `gt [3]` (transitions) | 15m | warning | +| `xrpld-node-state` | NodeNotFull | operating mode below FULL | `lt [4]` (FULL = 4) | 15m | warning | +| `xrpld-overlay` | ManifestJobQueueConvoy | `sum by (service_instance_id) (jobq_manifest_waiting)` | `gt [3]` (waiting jobs) | 10m | warning | +| `xrpld-overlay` | ManifestFloodInbound | inbound manifest byte rate | `gt [524288]` (B/s = 512 **KiB**/s, not 512 kB/s) | 10m | warning | +| `xrpld-overlay` | PeerResourceDisconnects | `sum by (service_instance_id) (increase(server_info{metric="peer_disconnects_resources"}[30m]))` | `gt [5]` | 5m | warning | + +**`ValidationsMissed` is a gated ratio, not `rate(...) > 0`.** The raw-rate shape +is the pre-fix version and it fires on **every non-validating node**: +`ValidationTracker` counts a miss whenever `weValidated && networkValidated` is +not both true, and a non-validator never sets `weValidated`, so its measured +ratio is exactly **1.0**. No threshold can separate "not a validator" from +"validator disagreeing", hence the `and on (...)` activity gate. The shipped +expression is: + +- numerator: `sum by (service_instance_id) (rate(validation_missed_total[15m]))` +- denominator: `clamp_min(` that same numerator `+ sum by (service_instance_id) (rate(validation_agreements_total[15m])), 1e-9)` +- gate: `and on (service_instance_id) (sum by (service_instance_id) (rate(validations_sent_total[15m])) > 0)` +- evaluator: `gt [0.1]` — i.e. >10% disagreement among nodes that do validate + +3 rules are `severity: critical`, 10 are `severity: warning`. + +Each rule carries labels `severity` and `category` and annotations `summary` + `description` (with `{{ $labels.service_instance_id }}` and `{{ $values.B.Value }}` interpolation). @@ -660,21 +898,31 @@ and `{{ $values.B.Value }}` interpolation). - **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not exact `0`) avoids float rate-noise suppressing the alert. -- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs - queued >1s at p99 indicate the node is saturated. -- Others are `> 0` on error/miss counters: any sustained nonzero rate is - actionable. +- **JobQueueLatencyHigh 1s p99**: `gt [1000000]` µs = 1s. A default starting + point, easy to tune — jobs queued >1s at p99 indicate the node is saturated. +- **ValidationsMissed `> 0.1` on a gated ratio**, not `> 0` on a raw rate: the + raw rate is permanently nonzero (ratio 1.0) on non-validators, so a `> 0` rule + pages on every non-validating node in the fleet. See the note above the + rationale list. +- **ManifestFloodInbound 524288 B/s**: an earlier 50 kB/s threshold produced ~41 + sustained 5-minute samples on healthy nodes; 512 KiB/s clears normal + manifest-exchange peaks. +- Remaining `gt [0]` rules (`LedgerHistoryMismatch`, `JobQueueTxOverflow`) sit on + true error counters where any sustained nonzero rate is actionable. ### Non-goals / YAGNI - No per-alert silencing schedules, no mute timings. -- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually). -- Single contact point — multi-receiver routing is left to the operator. +- No RPC/API or fee-market alerts (dashboards cover those visually). Overlay + alerts _were_ added during implementation — the `xrpld-overlay` group carries + three (manifest convoy, manifest flood, peer resource disconnects). +- Two contact points and a two-level policy tree shipped; deeper routing + (Discord, PagerDuty, per-team splits) is left to the operator. ### Verification 1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files. 2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses. 3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules` - returns the 6 rules; Grafana logs show no provisioning errors. + returns the 13 rules; Grafana logs show no provisioning errors. 4. Code-review pass (subagent) against phase conventions before commit. diff --git a/cfg/xrpld-example.cfg b/cfg/xrpld-example.cfg index 54d3936c42..f85f3bdcff 100644 --- a/cfg/xrpld-example.cfg +++ b/cfg/xrpld-example.cfg @@ -1627,8 +1627,18 @@ validators.txt # #------------------------------------------------------------------------------- # -# Enables distributed tracing via OpenTelemetry. Requires building with -# -DXRPL_ENABLE_TELEMETRY=ON (telemetry Conan option). +# Enables distributed tracing via OpenTelemetry. This section only has an +# effect if tracing was compiled in: build with CMake -Dtelemetry=ON (or Conan +# -o telemetry=True), and build it out with -Dtelemetry=OFF (or +# -o telemetry=False), which reduces all tracing code to no-ops. The option is +# currently ON so that CI compiles the telemetry code paths; OFF is the +# intended default once this feature is merged, so pass the value you want +# rather than relying on the default. +# +# Note that -DXRPL_ENABLE_TELEMETRY=OFF does NOT work: XRPL_ENABLE_TELEMETRY is +# a compile definition added by the build, not a CMake option, so it disables +# nothing. CMake only lists it at the end of configuration under +# "Manually-specified variables were not used by the project". # # [telemetry] # @@ -1647,12 +1657,29 @@ validators.txt # OTel resource attribute `service.instance.id`. Uniquely identifies # this node. Default: the node's public key (auto-detected). # +# SET THIS EXPLICITLY IF YOU USE THE METRICS PIPELINE. The node-public-key +# fallback only reaches traces: the metrics resource is built during +# startup, before the node key is known, and cannot be changed afterwards. +# With this key unset, metrics export with an empty service.instance.id and +# the per-node filter on the Grafana dashboards has nothing to split on. +# # endpoint=http://localhost:4318/v1/traces # -# The OTLP/HTTP exporter endpoint. The server sends trace data as -# protobuf-encoded HTTP POST requests to this URL. +# The OTLP/HTTP exporter endpoint for TRACES. The server sends trace data +# as protobuf-encoded HTTP POST requests to this URL. # Default: http://localhost:4318/v1/traces. # +# beast::insight metrics ([insight] server=otel) follow this setting: a +# trailing /v1/traces is rewritten to /v1/metrics. +# +# metrics_endpoint=http://localhost:4318/v1/metrics +# +# The OTLP/HTTP exporter endpoint for the internal metrics pipeline +# (the XRPL_METRIC_* instruments). This is a separate setting from +# `endpoint` and does NOT follow it, so a node exporting to a remote +# collector must set both. +# Default: http://localhost:4318/v1/metrics. +# # --- TLS settings for the OTLP exporter connection --- # # use_tls=0 diff --git a/docker/telemetry/.env.alerting.example b/docker/telemetry/.env.alerting.example index 594c55c064..aa0bdf9891 100644 --- a/docker/telemetry/.env.alerting.example +++ b/docker/telemetry/.env.alerting.example @@ -14,8 +14,8 @@ # GF_SMTP_* consumed by the Grafana container (compose `env_file`) to # turn on mail delivery. Without these, an email contact point # provisions fine and then silently sends nothing. -# ALERT_EMAIL_TO read by upload_alerts_to_grafana.py to build the Grafana -# CLOUD email contact point over the REST API (Cloud has no +# ALERT_EMAIL_TO the recipient for the Grafana CLOUD email contact point, +# which is created over the REST API (Cloud has no # provisioning filesystem). Not used by the local stack. # --- Slack (local stack: paste the webhook into contactpoints.yaml instead) --- @@ -25,7 +25,7 @@ SLACK_WEBHOOK_URL= # --- Email --- # Recipient for Grafana Cloud alerts (comma- or semicolon-separated). -# Consumed by upload_alerts_to_grafana.py. +# Used when creating the Cloud email contact point over the REST API. ALERT_EMAIL_TO= # SMTP relay Grafana sends through. Email only delivers when SMTP is enabled diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 7cf2b19fa0..048e6b8526 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -10,12 +10,16 @@ pipeline end-to-end, from span generation through the observability stack ### Build xrpld with telemetry +Follow [BUILD.md](../../BUILD.md) with `-o telemetry=True` added. From a build directory (`.build/`): + ```bash -conan install . --build=missing -o telemetry=True -cmake --preset default -Dtelemetry=ON -cmake --build --preset default --target xrpld +conan install .. --output-folder . --build missing -o telemetry=True --settings build_type=Release +cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Dxrpld=ON -Dtelemetry=ON .. +cmake --build . --target xrpld ``` +Conan also writes a `conan-release` preset, so `cmake --preset conan-release -Dtelemetry=ON` works too. There is no preset named `default`. + The binary is at `.build/xrpld`. ### Required tools @@ -46,13 +50,21 @@ docker compose -f docker/telemetry/docker-compose.yml up -d Wait for services to be ready: ```bash -# otel-collector health -curl -sf http://localhost:13133/ && echo "collector ready" +# otel-collector readiness: any HTTP response on the OTLP/HTTP port means the +# receiver is listening. Do NOT use `curl -sf` here — a GET of / returns 404, +# which -f treats as failure even when the collector is healthy. +[ "$(curl -so /dev/null -w '%{http_code}' http://localhost:4318/)" != "000" ] && + echo "collector ready" # Tempo readiness curl -sf http://localhost:3200/ready >/dev/null && echo "tempo ready" ``` +> The collector's `health_check` extension listens on **13133**, but +> `docker-compose.yml` publishes only 4317, 4318 and 8889 — so 13133 is not +> reachable from the host with the base stack. It is published only by the +> workload validation stack (`docker-compose.workload.yaml`). + ### Step 2: Start xrpld in standalone mode ```bash @@ -372,28 +384,62 @@ See the "Verification Queries" section below. ## Expected Span Catalog -All 16 production span names instrumented across Phases 2-5: +What follows is a **trigger** catalogue, not an attribute reference: one row per +span-name family, saying which config toggle gates it and what you have to do to +make it appear. It covers all 41 span-name families the code emits, in eight +subsystem groups — RPC (5), gRPC (1), Transaction (6), TxQ (6), Consensus (13), +Ledger (4), Peer (2), PathFind (4). -| Span Name | Source File | Phase | Key Attributes | How to Trigger | -| --------------------------- | ----------------- | ----- | ---------------------------------------------------------------------------------------- | ------------------------- | -| `rpc.http_request` | ServerHandler.cpp | 2 | -- | Any HTTP RPC call | -| `rpc.ws_upgrade` | ServerHandler.cpp | 2 | -- | WebSocket upgrade | -| `rpc.ws_message` | ServerHandler.cpp | 2 | -- | WebSocket RPC message | -| `rpc.process` | ServerHandler.cpp | 2 | -- | RPC processing | -| `rpc.command.` | RPCHandler.cpp | 2 | `command`, `version`, `rpc_role` | Any RPC command | -| `tx.process` | NetworkOPs.cpp | 3 | `xrpl.tx.hash`, `local`, `path` | Submit transaction | -| `tx.receive` | PeerImp.cpp | 3 | `xrpl.peer.id` | Peer relays transaction | -| `consensus.proposal.send` | RCLConsensus.cpp | 4 | `xrpl.consensus.round` | Consensus proposing phase | -| `consensus.ledger_close` | RCLConsensus.cpp | 4 | `xrpl.consensus.ledger.seq`, `xrpl.consensus.mode` | Ledger close event | -| `consensus.accept` | RCLConsensus.cpp | 4 | `xrpl.consensus.proposers`, `xrpl.consensus.round_time_ms` | Ledger accepted | -| `consensus.validation.send` | RCLConsensus.cpp | 4 | `xrpl.consensus.ledger.seq`, `xrpl.consensus.proposing` | Validation sent | -| `consensus.accept.apply` | RCLConsensus.cpp | 4 | `xrpl.consensus.close_time`, `close_time_correct`, `close_resolution_ms`, `state` | Ledger apply + close time | -| `tx.apply` | BuildLedger.cpp | 5 | `xrpl.ledger.tx_count`, `xrpl.ledger.tx_failed` | Ledger close (tx set) | -| `ledger.build` | BuildLedger.cpp | 5 | `xrpl.ledger.seq`, `xrpl.ledger.close_time`, `close_time_correct`, `close_resolution_ms` | Ledger build | -| `ledger.validate` | LedgerMaster.cpp | 5 | `xrpl.ledger.seq`, `xrpl.ledger.validations` | Ledger validated | -| `ledger.store` | LedgerMaster.cpp | 5 | `xrpl.ledger.seq` | Ledger stored | -| `peer.proposal.receive` | PeerImp.cpp | 5 | `xrpl.peer.id`, `proposal_trusted` | Peer sends proposal | -| `peer.validation.receive` | PeerImp.cpp | 5 | `xrpl.peer.id`, `validation_trusted` | Peer sends validation | +For each span's **attributes** — span name, source file, full attribute set and +description, per subsystem — see +[`docs/telemetry-runbook.md`](../../docs/telemetry-runbook.md) **§ Span +Reference**; its **§ Protocol Span Flow** gives the parent/child shape of a trace +and calls out where telemetry parenting deliberately differs from the protocol +flow. Both are kept in step with the code, so they are the reference to trust. +One hole worth knowing: the runbook's Span Reference tables have no row for +`grpc.` (it appears only in Protocol Span Flow). Its attributes are +`method`, `grpc_role` and `grpc_status`, emitted from `GRPCServer.cpp` with the +key constants in `src/xrpld/app/main/GrpcSpanNames.h`. + +If you find an older inline span inventory in this file or elsewhere, do not +trust it — the copy that used to live here had drifted badly (18 rows under a +"16 spans" heading, whole families missing, and pre-rename dotted `xrpl.*` +attribute keys the code no longer emits). The code and the runbook are the source +of truth. + +### Span → How to Trigger + +"Test" is the section of this file that exercises the family. `T1` = Test 1 +(standalone), `T2` = Test 2 (6-node network). + +| Span family (count) | Config toggle | How to trigger | Test | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| **RPC** (5 total, 3 here): `rpc.http_request`, `rpc.process`, `rpc.command.` | `trace_rpc=1` | Any HTTP JSON-RPC call: `curl -s http://localhost:5005 -d '{"method":"server_info"}'`. `rpc.command.` is one family — the command name is part of the span name. | T1 | +| **RPC** (cont.): `rpc.ws_message`, `rpc.ws_upgrade` | `trace_rpc=1` | Needs a WebSocket client against `[port_ws_public]` (**6005**) or `[port_ws_admin_local]` (6006). `rpc.ws_upgrade` covers the handshake — force a failure to see its error path. `curl` alone will not do it. | — | +| **gRPC** (1): `grpc.` | `trace_rpc=1` | Call a gRPC method (`GetLedger`, `GetLedgerData`, …). **Requires a `[port_grpc]` stanza — the shipped `xrpld-telemetry*.cfg` files define none**, so add one first. | — | +| **Transaction** (6 total, 4 here): `tx.process`, `tx.preflight`, `tx.preclaim`, `tx.transactor` | `trace_transactions` | Submit any transaction (T1 Step 4). The three apply-stage spans share the tx's deterministic trace id; the `stage` attribute says where a failing tx stopped. | T1 | +| **Transaction** (cont.): `tx.receive` | `trace_transactions` | A **peer** relays a transaction. Never appears in standalone — submit on one node of the cluster and look on another. | T2 | +| **Transaction** (cont.): `tx.apply` | `trace_transactions` | Ledger close with a non-empty transaction set: submit, then `ledger_accept` (T1) or wait for consensus (T2). | T1 / T2 | +| **TxQ** (6): `txq.enqueue`, `txq.apply_direct`, `txq.batch_clear`, `txq.accept`, `txq.accept_tx`, `txq.cleanup` | `trace_transactions` | `txq.enqueue`/`apply_direct` on every submission; `txq.accept`/`accept_tx`/`cleanup` on every ledger close. To force real queueing, submit faster than ledgers close or with a fee below the required fee level. | T1 | +| **Consensus** (13): `consensus.round`, `.phase.open`, `.establish`, `.update_positions`, `.check`, `.proposal.send`, `.ledger_close`, `.accept`, `.accept.apply`, `.validation.send`, `.mode_change`, `.proposal.receive`, `.validation.receive` | `trace_consensus=1` | Requires real consensus — **standalone emits none of these**. Bring up T2 and wait for nodes to reach `proposing`; one `consensus.round` per close. `.mode_change` needs an actual mode transition (stop/start a node). | T2 | +| **Ledger** (4 total, 3 here): `ledger.build`, `ledger.validate`, `ledger.store` | `trace_ledger=1` | Any ledger close: `ledger_accept` in standalone, or consensus in T2. | T1 / T2 | +| **Ledger** (cont.): `ledger.acquire` | `trace_ledger=1` | Node fetches a **missing** ledger from peers. Start a node with no history against a running cluster, or restart one node after the others have advanced. | T2 | +| **Peer** (2): `peer.proposal.receive`, `peer.validation.receive` | `trace_peer=1` | Inbound consensus messages from peers; fresh trace roots. T2 only, and high volume. | T2 | +| **PathFind** (4): `pathfind.request`, `pathfind.compute`, `pathfind.discover`, `pathfind.update_all` | `trace_rpc=1` | `curl -s http://localhost:5005 -d '{"method":"ripple_path_find","params":[{"source_account":"…","destination_account":"…","destination_amount":"100"}]}'`. `pathfind.update_all` fires on ledger close while a request is active. | T1 | + +Notes that matter when a span you expect is missing: + +- **Toggles are per-subsystem and all default to on** (`trace_rpc`, + `trace_transactions`, `trace_consensus`, `trace_peer`, `trace_ledger`), but + `[telemetry] enabled` defaults to **0** — nothing is emitted until it is `1`. +- **`consensus.*` and `peer.*` cannot be produced in standalone mode.** If Test 1 + shows none, that is correct behaviour, not a regression — see "Expected spans + (standalone mode)" above. +- **`rpc.ws_*` and `grpc.*` need a client and a port the quick tests do not + use.** Absence in T1/T2 is expected. +- Trace ids are deterministic for transactions (`txID[0:16]`) and consensus + rounds (`prevLedgerHash[0:16]`), so you can compute the id you expect rather + than searching for it. --- @@ -456,13 +502,17 @@ curl -s "$PROM/api/v1/query?query=span_calls_total" | Open http://localhost:3000 (anonymous admin access enabled). -Pre-configured dashboards: +Pre-configured dashboards: every `.json` under +`docker/telemetry/grafana/dashboards/` is provisioned into the `xrpld` folder — +`provisioning/dashboards/dashboards.yaml` points the file provider at +`/var/lib/grafana/dashboards`, which `docker-compose.yml` bind-mounts from that +directory. Adding a file there is all that is needed; there is no per-dashboard +registration. -- **RPC Performance**: Request rates, latency percentiles by command, top commands, WebSocket rate -- **Transaction Overview**: Transaction processing rates, apply duration, peer relay, failed tx rate -- **Consensus Health**: Consensus round duration, proposer counts, mode tracking, accept heatmap -- **Ledger Operations**: Build/validate/store rates and durations, TX apply metrics -- **Peer Network**: Proposal/validation receive rates, trusted vs untrusted breakdown (requires `trace_peer=1`) +For what each dashboard covers, see +[`docs/telemetry-runbook.md`](../../docs/telemetry-runbook.md) **§ Grafana +Dashboards** — the per-dashboard reference. Listing them here would be a second +copy that rots (this section previously named 5 of the 15 provisioned). Pre-configured datasources: @@ -504,10 +554,35 @@ docker compose -f docker/telemetry/docker-compose.yml \ -f docker/telemetry/docker-compose.grafanacloud.yaml up -d ``` -The override swaps the collector onto `otel-collector-config.grafanacloud.yaml`, -which keeps the local Tempo/Prometheus/Loki exporters and adds one -OTLP/HTTP exporter to Grafana Cloud on all three pipelines. Bring the stack -up with just the base file to return to local-only. +The override swaps the collector onto `otel-collector-config.grafanacloud.yaml`. +It keeps the local Tempo/Prometheus/Loki exporters and adds an +`otlphttp/grafanacloud` exporter, but it is **not** the base config plus one +exporter — it restructures the pipelines. Bring the stack up with just the base +file to return to local-only. + +Differences that change what you will see: + +| | Base (`otel-collector-config.yaml`) | Cloud override | +| ------------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------- | +| Pipelines | 3: `traces`, `metrics`, `logs` | 5: `traces/metrics`, `traces/store`, `metrics/local`, `metrics/cloud`, `logs` | +| Trace sampling | none — 100% of spans reach Tempo | `tail_sampling` keeps **0.5%** (one `probabilistic` policy, `decision_wait: 10s`) on `traces/store` | +| `debug` exporter | present on `traces` | dropped | +| `attributes/hash` | present on `traces` | **omitted** | +| Cloud metric labels | n/a | `transform/cloudlabels` on `metrics/cloud` only | + +Consequences worth knowing before you debug against the cloud stack: + +- **Traces are sampled, span metrics are not.** Sampling sits only on + `traces/store` (the pipeline feeding Tempo _and_ Grafana Cloud). The + `spanmetrics` connector is fed by the separate, unsampled `traces/metrics` + pipeline, so `span_*` rates stay exact while only ~1 trace in 200 is + retrievable by trace ID. A trace you can see in a metric may not exist in + Tempo. +- **Pathfinding account hashing does not happen on the cloud export.** The base + config's `attributes/hash` processor hashes `pathfind_source_account` and + `pathfind_dest_account`. It is absent from every cloud pipeline, so those two + attributes leave for Grafana Cloud (and, on that config, for Tempo) with their + raw account values. ### Step 4: Verify data reaches Grafana Cloud @@ -516,7 +591,7 @@ Cloud instance and confirm: - **Traces**: Explore → hosted Tempo datasource → search `{resource.service.name="xrpld"}` - **Metrics**: Explore → hosted Prometheus/Mimir → query `span_calls_total` -- **Logs**: Explore → hosted Loki → query `{job="xrpld"}` (requires `warning`+ file logging) +- **Logs**: Explore → hosted Loki → query `{service_name="xrpld"}` (requires `warning`+ file logging). **Not `{job="xrpld"}`** — see the note under Test 3 Step 3. If nothing appears, check the collector logs for auth/export errors: @@ -531,9 +606,9 @@ means the endpoint URL is wrong or missing the `/otlp` path. --- -## Test 3: Log-Trace Correlation (Phase 8) +## Test 3: Log-Trace Correlation -Phase 8 injects `trace_id` and `span_id` into xrpld's log output when +xrpld injects `trace_id` and `span_id` into its log output when a log line is emitted within an active OTel span. This test verifies the end-to-end log-trace correlation pipeline. @@ -576,12 +651,28 @@ exports parsed entries to Loki. Verify Loki has received entries: ```bash # Query Loki for any xrpld logs curl -sG "http://localhost:3100/loki/api/v1/query" \ - --data-urlencode 'query={job="xrpld"}' \ + --data-urlencode 'query={service_name="xrpld"}' \ --data-urlencode 'limit=5' | jq '.data.result | length' ``` Expected: > 0 results. +> **Use `service_name`, not `job`.** The collector's `resource/logs` processor +> applies an `upsert` to **both** `service.name=xrpld` and `job=xrpld` +> (`otel-collector-config.yaml:57-70`), and its comment says the `job` attribute +> is there so operators can paste `{job="xrpld"}`. That does not work: on OTLP +> ingest Loki promotes only an allow-listed set of resource attributes to indexed +> stream labels (`service.name` → `service_name`, plus `service.namespace`, +> `service.instance.id`, `deployment.environment`, `k8s.*`, `cloud.*`), and `job` +> is not on the list. This repo mounts no Loki config override — the `loki` +> service runs the image's built-in `/etc/loki/local-config.yaml` +> (`docker-compose.yml:75`) — so `job` lands in **structured metadata**, which +> cannot be a stream selector. `{job="xrpld"}` therefore returns **zero results +> with no error**, which reads exactly like "logs are not being ingested". If +> this query is empty, check `{service_name="xrpld"}` before debugging the +> pipeline. All 38 Loki queries in the shipped dashboards select on +> `service_name`; none uses `job`. + ### Step 4: Verify Grafana Tempo-to-Loki correlation 1. Open Grafana at http://localhost:3000 @@ -593,7 +684,7 @@ Expected: > 0 results. ### Step 5: Verify Grafana Loki-to-Tempo correlation 1. In Grafana **Explore**, select **Loki** datasource -2. Query: `{job="xrpld"} |= "trace_id="` +2. Query: `{service_name="xrpld"} |= "trace_id="` 3. In the log results, click the **TraceID** derived field link 4. Verify it navigates to the full trace in Tempo @@ -620,9 +711,10 @@ Expected: > 0 results. docker compose -f docker/telemetry/docker-compose.yml logs otel-collector ``` 2. Verify xrpld telemetry config has `enabled=1` and correct endpoint -3. Check that otel-collector port 4318 is accessible: +3. Check that otel-collector port 4318 is accessible (`-f` would fail on the + receiver's 404 for `GET /`, so test for any HTTP status instead): ```bash - curl -sf http://localhost:4318 && echo "reachable" + curl -so /dev/null -w '%{http_code}\n' http://localhost:4318/ ``` 4. Increase `batch_delay_ms` or decrease `batch_size` in xrpld config @@ -650,7 +742,7 @@ Expected: > 0 results. 2. Check submit response for error codes 3. In standalone mode, remember to call `ledger_accept` after submitting -### No trace_id in log output (Phase 8) +### No trace_id in log output 1. Verify xrpld was built with `telemetry=ON` (`-Dtelemetry=ON` in CMake) 2. Verify `enabled=1` in the `[telemetry]` config section @@ -659,7 +751,7 @@ Expected: > 0 results. `trace_id`/`span_id`. 4. Ensure the trace category is enabled (e.g., `trace_rpc=1` for RPC logs) -### No logs in Loki (Phase 8) +### No logs in Loki 1. Verify the log file mount in docker-compose.yml: ```yaml @@ -680,7 +772,7 @@ Expected: > 0 results. 4. Verify the filelog receiver glob pattern matches your log files: The default pattern is `/var/log/xrpld/*/debug.log` -### Grafana trace-log links not working (Phase 8) +### Grafana trace-log links not working 1. Verify `tracesToLogs` is configured in the Tempo datasource provisioning (`docker/telemetry/grafana/provisioning/datasources/tempo.yaml`) @@ -694,14 +786,21 @@ Expected: > 0 results. ### Spanmetrics not appearing in Prometheus 1. Verify otel-collector config has `spanmetrics` connector -2. Check that the metrics pipeline is configured: +2. Check that the metrics pipeline matches `otel-collector-config.yaml` + verbatim: ```yaml service: pipelines: metrics: - receivers: [spanmetrics] + receivers: [otlp, spanmetrics] + processors: [resource/tier, resource/stripsdk, batch] exporters: [prometheus] ``` + Both receivers are required. `spanmetrics` carries the span-derived + `span_*` series; `otlp` carries the node's native `beast::insight` / + MetricsRegistry metrics, which arrive on the same OTLP port. Dropping + `otlp` silently removes every native metric while the `span_*` ones keep + working — so the dashboards only half-break. 3. Verify Prometheus can reach collector: ```bash curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets' diff --git a/docker/telemetry/docker-compose.workload.yaml b/docker/telemetry/docker-compose.workload.yaml index 818c65f4c5..6ab809d705 100644 --- a/docker/telemetry/docker-compose.workload.yaml +++ b/docker/telemetry/docker-compose.workload.yaml @@ -1,24 +1,32 @@ -# Docker Compose workload harness for Phase 10 telemetry validation. +# Docker Compose workload harness for telemetry validation. # -# Runs a 5-node validator cluster with full OTel telemetry stack: -# - 5 rippled validator nodes (consensus network) +# Runs the OTel telemetry backend only. There are no validator services here: # - OTel Collector (traces + native OTLP metrics) # - Tempo (trace backend + search API) # - Prometheus (metrics) # - Loki (log aggregation for log-trace correlation) # - Grafana (dashboards + trace/log exploration) # +# The validator cluster runs as host processes, not containers. +# run-full-validation.sh starts NUM_NODES (default 5) xrpld instances on +# 127.0.0.1, each with a cfg it generates inline, peered to each other via +# [ips_fixed]. They reach the collector through the published ports below and +# write their logs into the bind-mounted workdir for the filelog receiver. +# # Usage: -# # Start the harness (requires pre-built xrpld image or mount binary): +# # Start the telemetry backend on its own: # docker compose -f docker/telemetry/docker-compose.workload.yaml up -d # -# # Or use the orchestrator: +# # Or let the orchestrator start this stack and the node cluster together: # docker/telemetry/workload/run-full-validation.sh # -# Prerequisites: +# Prerequisites (for the orchestrator, not for this stack): # - xrpld binary built with -DXRPL_ENABLE_TELEMETRY=ON # - Validator keys generated via generate-validator-keys.sh -# - Node configs generated by run-full-validation.sh +# +# Image tags are pinned to the same versions as docker-compose.yml, which +# mounts these same collector, Tempo and Prometheus config files. Floating +# tags would let an upstream release change the harness result. # # Note: No Docker healthchecks are defined here. The orchestrator script # (run-full-validation.sh) polls each service endpoint directly from the @@ -30,7 +38,7 @@ services: # --------------------------------------------------------------------------- otel-collector: - image: otel/opentelemetry-collector-contrib:latest + image: otel/opentelemetry-collector-contrib:0.158.0 command: ["--config=/etc/otel-collector-config.yaml"] ports: - "4317:4317" # OTLP gRPC @@ -49,7 +57,7 @@ services: - workload-net tempo: - image: grafana/tempo:2.7.2 + image: grafana/tempo:2.9.4 command: ["-config.file=/etc/tempo.yaml"] ports: - "3200:3200" # Tempo HTTP API @@ -60,7 +68,7 @@ services: - workload-net prometheus: - image: prom/prometheus:latest + image: prom/prometheus:v3.13.2 ports: - "9090:9090" volumes: @@ -71,7 +79,7 @@ services: - workload-net loki: - image: grafana/loki:3.4.2 + image: grafana/loki:3.7.6 ports: - "3100:3100" # Loki HTTP API command: ["-config.file=/etc/loki/local-config.yaml"] @@ -79,12 +87,18 @@ services: - workload-net grafana: - image: grafana/grafana:latest + image: grafana/grafana:13.1.2 + # Anonymous Admin is deliberate, and matches the sibling stack in + # docker-compose.yml. This stack is an ephemeral local/CI backend that + # run-full-validation.sh brings up and tears down around a single run; it + # holds no durable data and is never exposed beyond the host. Admin rather + # than Viewer because the harness drives the Grafana API against it, and + # the dashboards and datasources are provisioned from the mounts below. environment: - - GF_AUTH_ANONYMOUS_ENABLED=true - - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin + - GF_AUTH_ANONYMOUS_ENABLED=true # No login required for local dev + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin # Full access without auth ports: - - "3000:3000" + - "3000:3000" # Grafana web UI volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro diff --git a/docker/telemetry/docker-compose.yml b/docker/telemetry/docker-compose.yml index 8aa2c3df64..b39e60ce43 100644 --- a/docker/telemetry/docker-compose.yml +++ b/docker/telemetry/docker-compose.yml @@ -8,10 +8,16 @@ # - tempo: Grafana Tempo tracing backend, queryable via Grafana Explore # on port 3000. Recommended for production (S3/GCS storage, TraceQL). # - loki: Grafana Loki log aggregation backend for centralized log -# ingestion and log-trace correlation (Phase 8). +# ingestion and log-trace correlation. # - grafana: dashboards on port 3000, pre-configured with Tempo, # Prometheus, and Loki datasources. # +# Requires Docker Compose >= 2.24.0. The grafana service uses the long-form +# `env_file` mapping (`path:` / `required:`), which older Compose cannot parse — +# and it fails for the whole file, not just that service. The long form is +# needed: `.env.alerting` is gitignored and absent in a fresh clone, and the +# short form treats a missing env file as an error. +# # Usage: # docker compose -f docker/telemetry/docker-compose.yml up -d # @@ -21,11 +27,37 @@ # endpoint=http://localhost:4318/v1/traces services: + # One-shot init for the collector's offset store. Docker creates a fresh + # named volume owned by root, but the collector image runs as 10001:10001 + # and ships no writable directory, so the file_storage extension could not + # create its database and the collector would fail to start. Chown the + # volume once, then exit; the collector waits for this to complete. + # + # Reuses the Prometheus image purely because the stack already pulls it and + # it has a shell — this adds no new image dependency. The entrypoint is + # overridden since that image normally starts the Prometheus server. + otelcol-storage-init: + image: prom/prometheus:v3.13.2 + user: "0:0" + entrypoint: ["sh", "-c"] + command: ["mkdir -p /data/file_storage && chown -R 10001:10001 /data"] + volumes: + - otelcol-storage:/data + networks: + - xrpld-telemetry + # OpenTelemetry Collector: receives spans from xrpld via OTLP protocol, # batches them for efficiency, and forwards to Tempo for storage. otel-collector: image: otel/opentelemetry-collector-contrib:0.158.0 - command: ["--config=/etc/otel-collector-config.yaml"] + # Second --config layers filelog offset persistence on top of the shared + # base config; the collector deep-merges them. Only this stack keeps its + # logs across restarts, so only this stack needs it. + command: + [ + "--config=/etc/otel-collector-config.yaml", + "--config=/etc/otel-collector-filestorage.yaml", + ] ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP (traces + native OTel metrics) @@ -36,6 +68,8 @@ services: volumes: # Mount collector pipeline config (receivers → processors → exporters) - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro + # Dev-only overlay: persist filelog read offsets across restarts + - ./otel-collector-filestorage.yaml:/etc/otel-collector-filestorage.yaml:ro # Mount the xrpld log root for the filelog receiver. The telemetry # configs write to docker/telemetry/data/logs//debug.log, so # the default source is the repo-relative ./data/logs — user-owned and @@ -43,9 +77,16 @@ services: # XRPLD_LOG_DIR to point at another root (e.g. the integration test sets # it to its own workdir). Mounted read-only so the collector only tails. - ${XRPLD_LOG_DIR:-./data/logs}:/var/log/xrpld:ro + # Persisted filelog read offsets, so a collector restart resumes + # instead of re-reading every debug.log from the top. + - otelcol-storage:/var/lib/otelcol depends_on: - - tempo - - loki + tempo: + condition: service_started + loki: + condition: service_started + otelcol-storage-init: + condition: service_completed_successfully networks: - xrpld-telemetry @@ -156,6 +197,7 @@ volumes: tempo-data: prometheus-data: loki-data: + otelcol-storage: # Isolated bridge network so services communicate by container name # (e.g., the collector reaches Tempo at http://tempo:4317). diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 00579d9736..2dc19084e3 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -61,7 +61,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.send\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.send\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -99,7 +99,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (consensus_mode, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", consensus_mode=~\"$consensus_mode\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"$1\", \"consensus_mode\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (consensus_mode, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", consensus_mode=~\"$consensus_mode\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"$1\", \"consensus_mode\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -157,7 +157,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.proposal.send\"}[$__rate_interval])), \"series\", \"Proposals / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.proposal.send\"}[$__rate_interval])), \"series\", \"Proposals / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -202,14 +202,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\"}[$__rate_interval])), \"series\", \"Accepts / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\"}[$__rate_interval])), \"series\", \"Accepts / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"Closes / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"Closes / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -254,14 +254,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.send\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.send\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"Closes / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.ledger_close\"}[$__rate_interval])), \"series\", \"Closes / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -350,6 +350,7 @@ "x": 0, "y": 41 }, + "options": { "tooltip": { "mode": "multi", @@ -663,7 +664,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.update_positions\"}[5m]))), \"series\", \"P95 Update\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.update_positions\"}[5m]))), \"series\", \"P95 Update\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -708,7 +709,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", consensus_mode=~\"$consensus_mode\", span_name=\"consensus.round\"}[5m]))), \"series\", \"P95 Close (consensus.round)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", consensus_mode=~\"$consensus_mode\", span_name=\"consensus.round\"}[5m]))), \"series\", \"P95 Close (consensus.round)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -746,7 +747,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept.apply\"}[5m]))), \"series\", \"P95 Apply Duration\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept.apply\"}[5m]))), \"series\", \"P95 Apply Duration\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -840,7 +841,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (close_time_correct, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{span_name=\"consensus.accept.apply\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"close_time_correct\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (close_time_correct, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{span_name=\"consensus.accept.apply\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"close_time_correct\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -855,7 +856,8 @@ "pointSize": 5, "lineWidth": 1, "fillOpacity": 0, - "gradientMode": "none" + "gradientMode": "none", + "axisLabel": "NetClock Seconds (Ripple Epoch)" } }, "overrides": [] @@ -897,8 +899,8 @@ "overrides": [ { "matcher": { - "id": "byName", - "options": "Vote Bins" + "id": "byRegexp", + "options": ".*vote_bins.*" }, "properties": [ { @@ -913,8 +915,8 @@ }, { "matcher": { - "id": "byName", - "options": "Resolution" + "id": "byRegexp", + "options": ".*close_resolution_ms.*" }, "properties": [ { @@ -1005,7 +1007,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, consensus_state, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"consensus_state\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, consensus_state, xrpl_branch, xrpl_node_role, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"consensus_state\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1040,14 +1042,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state=\"moved_on\"}[$__rate_interval])), \"series\", \"moved_on\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state=\"moved_on\"}[$__rate_interval])), \"series\", \"moved_on\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state=\"expired\"}[$__rate_interval])), \"series\", \"expired\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.accept\", consensus_state=\"expired\"}[$__rate_interval])), \"series\", \"expired\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1092,14 +1094,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.check\", consensus_stalled=\"true\"}[$__rate_interval])), \"series\", \"Stalled\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.check\", consensus_stalled=\"true\"}[$__rate_interval])), \"series\", \"Stalled\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.check\", consensus_stalled=\"false\"}[$__rate_interval])), \"series\", \"Not Stalled\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.check\", consensus_stalled=\"false\"}[$__rate_interval])), \"series\", \"Not Stalled\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1144,7 +1146,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (mode_new, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.mode_change\"}[$__rate_interval])), \"series\", \"$1\", \"mode_new\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (mode_new, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.mode_change\"}[$__rate_interval])), \"series\", \"$1\", \"mode_new\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1189,7 +1191,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role) (rate(ledger_history_mismatch_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(ledger_history_mismatch_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/fee-market.json b/docker/telemetry/grafana/dashboards/fee-market.json index 2d572f454c..71dd0ba6ed 100644 --- a/docker/telemetry/grafana/dashboards/fee-market.json +++ b/docker/telemetry/grafana/dashboards/fee-market.json @@ -427,7 +427,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(txq_expired_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Expired / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(txq_expired_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Expired / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -472,7 +472,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role) (rate(txq_dropped_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(txq_dropped_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index e6e3fae3ad..5b2d4ed048 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -1225,42 +1225,42 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_ledgerdata_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"ledgerdata q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_ledgerdata_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"ledgerdata q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_acceptledger_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"acceptledger q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_acceptledger_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"acceptledger q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_fetchtxndata_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"fetchtxndata q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_fetchtxndata_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"fetchtxndata q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_transaction_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"transaction q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_transaction_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"transaction q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_advanceledger_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"advanceledger q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_advanceledger_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"advanceledger q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(jobq_ledgerrequest_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"ledgerrequest q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(jobq_ledgerrequest_q_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"ledgerrequest q-wait p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1305,7 +1305,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_duration_us\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"NuDB us/read\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_duration_us\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"NuDB us/read\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1350,7 +1350,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(ios_latency_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"IO Service p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(ios_latency_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"IO Service p95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1395,7 +1395,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_hit\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"NuDB Found Ratio\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_hit\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"NuDB Found Ratio\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1703,7 +1703,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, handler, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"ledgerRequest\", handler=~\"$handler\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"$1 q-wait p99\", \"handler\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, handler, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"ledgerRequest\", handler=~\"$handler\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"$1 q-wait p99\", \"handler\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -1768,7 +1768,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_duration_us\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"Read Mean (Windowed)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_duration_us\"}[$__rate_interval])) / (sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])) > 0), \"series\", \"Read Mean (Windowed)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index 27ca03f666..9cc7726cb0 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -74,7 +74,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[$__rate_interval])), \"series\", \"Builds / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[$__rate_interval])), \"series\", \"Builds / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -112,7 +112,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.validate\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.validate\"}[$__rate_interval])), \"series\", \"Validations / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -150,7 +150,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[5m]))), \"series\", \"P95 Build Duration\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[5m]))), \"series\", \"P95 Build Duration\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -257,7 +257,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[5m]))), \"series\", \"P95 tx.apply\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[5m]))), \"series\", \"P95 tx.apply\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -302,7 +302,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[$__rate_interval])), \"series\", \"tx.apply / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[$__rate_interval])), \"series\", \"tx.apply / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -360,7 +360,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.store\"}[$__rate_interval])), \"series\", \"Stores / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.store\"}[$__rate_interval])), \"series\", \"Stores / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -398,14 +398,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[5m]))), \"series\", \"P95 ledger.build\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.build\"}[5m]))), \"series\", \"P95 ledger.build\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.round\"}[5m]))), \"series\", \"P95 Close (consensus.round)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.round\"}[5m]))), \"series\", \"P95 Close (consensus.round)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -457,7 +457,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(1 / sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(ledgers_closed_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Close Interval\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(1 / sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(ledgers_closed_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Close Interval\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/network-traffic.json b/docker/telemetry/grafana/dashboards/network-traffic.json index aa2ed62610..f258ff8db1 100644 --- a/docker/telemetry/grafana/dashboards/network-traffic.json +++ b/docker/telemetry/grafana/dashboards/network-traffic.json @@ -482,7 +482,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(topk(10, label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ping_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ping_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(status_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"status_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(havetxset_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"havetxset_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledgerdata_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledgerdata_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\")), \"series\", \"$1\", \"__name__\", \"(.*)\")" + "expr": "label_replace(topk(10, label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ping_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ping_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(status_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"status_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(havetxset_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"havetxset_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledgerdata_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledgerdata_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\")), \"series\", \"$1\", \"__name__\", \"(.*)\")" } ], "fieldConfig": { @@ -877,7 +877,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(topk(15, label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_cas_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_cas_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_cas_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_cas_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_fetch_pack_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_fetch_pack_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_transaction_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_transaction_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(getobject_transactions_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transactions_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(have_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"have_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_data_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_cluster_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_cluster_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_manifest_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_manifest_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proof_path_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proof_path_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proof_path_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proof_path_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proposals_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(proposals_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(replay_delta_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"replay_delta_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(replay_delta_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"replay_delta_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(requested_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"requested_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(set_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(squelch_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(squelch_ignored_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_ignored_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(squelch_suppressed_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_suppressed_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(transactions_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(unknown_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"unknown_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(validations_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(validations_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role)(rate(validator_lists_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validator_lists_bytes_in\",\"\",\"\")), \"series\", \"$1\", \"__name__\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(topk(15, label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_cas_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_cas_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_cas_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_cas_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_fetch_pack_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_fetch_pack_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_transaction_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_transaction_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(getobject_transactions_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"getobject_transactions_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(have_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"have_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_data_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_cluster_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_cluster_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_manifest_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_manifest_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proof_path_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proof_path_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proof_path_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proof_path_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proposals_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(proposals_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"proposals_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(replay_delta_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"replay_delta_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(replay_delta_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"replay_delta_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(requested_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"requested_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(set_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"set_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(squelch_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(squelch_ignored_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_ignored_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(squelch_suppressed_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"squelch_suppressed_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(transactions_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"transactions_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(unknown_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"unknown_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(validations_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(validations_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validations_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item)(rate(validator_lists_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])),\"__name__\",\"validator_lists_bytes_in\",\"\",\"\")), \"series\", \"$1\", \"__name__\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index c2dc0e5b32..9a21fca75c 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -574,7 +574,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(ios_latency_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 I/O Latency\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(ios_latency_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 I/O Latency\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], @@ -810,7 +810,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative count of transitions into each operating mode.*\n\n###### How it's computed:\n*Current value of each per-mode transition counter, plotted as lines.*\n\n###### Reading it:\n*Flat lines are healthy; steps up mean the node changed mode.*\n\n###### Healthy range:\n*Few transitions once the node is stable in Full mode.*\n\n###### Watch for:\n*Frequent transitions out of Full, or into Disconnected or Syncing, indicate instability.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::Stats`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Transitions into each operating mode, per interval.*\n\n###### How it's computed:\n*increase() over each per-mode transition counter, so each point is the number of transitions in that bucket and the series stays correct across an xrpld restart (the counters reset to 0).*\n\n###### Reading it:\n*Zero is healthy; each point is a mode change within that bucket. Brief flaps show up here even when they are too short to appear on Operating Mode (State Timeline), which can only sample state once per scrape.*\n\n###### Healthy range:\n*Few transitions once the node is stable in Full mode.*\n\n###### Watch for:\n*Frequent transitions out of Full, or into Disconnected or Syncing, indicate instability.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::Stats`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "fieldConfig": { "defaults": { "color": { @@ -901,7 +901,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(state_accounting_full_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Full\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(increase(state_accounting_full_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Full\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" }, { @@ -909,7 +909,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(state_accounting_tracking_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Tracking\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(increase(state_accounting_tracking_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Tracking\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" }, { @@ -917,7 +917,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(state_accounting_syncing_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Syncing\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(increase(state_accounting_syncing_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Syncing\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "C" }, { @@ -925,7 +925,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(state_accounting_connected_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Connected\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(increase(state_accounting_connected_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Connected\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "D" }, { @@ -933,7 +933,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(state_accounting_disconnected_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Disconnected\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(increase(state_accounting_disconnected_transitions{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Disconnected\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "E" } ], @@ -2100,7 +2100,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"acceptLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Accept Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"acceptLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Accept Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" }, { @@ -2108,7 +2108,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"advanceLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Advance Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"advanceLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Advance Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" }, { @@ -2116,7 +2116,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"transaction\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Transaction\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"transaction\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Transaction\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "C" }, { @@ -2124,7 +2124,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"writeObjects\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Write Objects\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"writeObjects\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Write Objects\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "D" }, { @@ -2132,7 +2132,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"heartbeat\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Heartbeat\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"heartbeat\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Heartbeat\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "E" }, { @@ -2140,7 +2140,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"sweep\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Sweep\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"sweep\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Sweep\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "F" }, { @@ -2148,7 +2148,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"trustedValidation\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Validation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"trustedValidation\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Validation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "G" }, { @@ -2156,7 +2156,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"trustedProposal\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Proposal\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"trustedProposal\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Proposal\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "H" }, { @@ -2164,7 +2164,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"publishNewLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Publish New Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"publishNewLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Publish New Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "I" }, { @@ -2172,7 +2172,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"clientRPC\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Client RPC\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"clientRPC\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Client RPC\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "J" }, { @@ -2180,7 +2180,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{job_type=\"ledgerData\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Ledger Data\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{job_type=\"ledgerData\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Ledger Data\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "K" } ], @@ -2283,7 +2283,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"acceptLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Accept Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"acceptLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Accept Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" }, { @@ -2291,7 +2291,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"advanceLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Advance Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"advanceLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Advance Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" }, { @@ -2299,7 +2299,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"transaction\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Transaction\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"transaction\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Transaction\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "C" }, { @@ -2307,7 +2307,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"writeObjects\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Write Objects\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"writeObjects\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Write Objects\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "D" }, { @@ -2315,7 +2315,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"heartbeat\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Heartbeat\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"heartbeat\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Heartbeat\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "E" }, { @@ -2323,7 +2323,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"sweep\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Sweep\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"sweep\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Sweep\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "F" }, { @@ -2331,7 +2331,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"trustedValidation\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Validation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"trustedValidation\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Validation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "G" }, { @@ -2339,7 +2339,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"trustedProposal\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Proposal\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"trustedProposal\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Trusted Proposal\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "H" }, { @@ -2347,7 +2347,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"publishNewLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Publish New Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"publishNewLedger\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Publish New Ledger\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "I" }, { @@ -2355,7 +2355,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"clientRPC\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Client RPC\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"clientRPC\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Client RPC\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "J" }, { @@ -2363,7 +2363,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{job_type=\"ledgerData\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Ledger Data\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{job_type=\"ledgerData\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"Ledger Data\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "K" } ], @@ -2466,7 +2466,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (service_instance_id, le, job_type, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"$1\", \"job_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (service_instance_id, le, job_type, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"$1\", \"job_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], @@ -2569,7 +2569,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (service_instance_id, le, job_type, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"$1\", \"job_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(histogram_quantile($quantile, sum by (service_instance_id, le, job_type, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"$1\", \"job_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], @@ -3017,7 +3017,7 @@ "refId": "A" } ], - "title": "FullBelowCache Hit Rate", + "title": "FullBelowCache Hit Rate [$xrpl_network_type]", "type": "gauge" } ], @@ -3504,9 +3504,9 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "max by (xrpl_network_type) (server_info{metric=\"validated_ledger_seq\",service_name=~\"$service_name\",service_instance_id=~\"$node\",xrpl_network_type=~\"$xrpl_network_type\",xrpl_branch=~\"$xrpl_branch\",xrpl_node_role=~\"$xrpl_node_role\",deployment_environment=~\"$deployment_environment\",xrpl_work_item=~\"$xrpl_work_item\"}) - min by (xrpl_network_type) (server_info{metric=\"validated_ledger_seq\",service_name=~\"$service_name\",service_instance_id=~\"$node\",xrpl_network_type=~\"$xrpl_network_type\",xrpl_branch=~\"$xrpl_branch\",xrpl_node_role=~\"$xrpl_node_role\",deployment_environment=~\"$deployment_environment\",xrpl_work_item=~\"$xrpl_work_item\"})", + "expr": "max by (xrpl_network_type, xrpl_work_item) (server_info{metric=\"validated_ledger_seq\",service_name=~\"$service_name\",service_instance_id=~\"$node\",xrpl_network_type=~\"$xrpl_network_type\",xrpl_branch=~\"$xrpl_branch\",xrpl_node_role=~\"$xrpl_node_role\",deployment_environment=~\"$deployment_environment\",xrpl_work_item=~\"$xrpl_work_item\"}) - min by (xrpl_network_type, xrpl_work_item) (server_info{metric=\"validated_ledger_seq\",service_name=~\"$service_name\",service_instance_id=~\"$node\",xrpl_network_type=~\"$xrpl_network_type\",xrpl_branch=~\"$xrpl_branch\",xrpl_node_role=~\"$xrpl_node_role\",deployment_environment=~\"$deployment_environment\",xrpl_work_item=~\"$xrpl_work_item\"})", "instant": true, - "legendFormat": "Spread [{{xrpl_network_type}}]", + "legendFormat": "Spread [{{xrpl_network_type}}] [{{xrpl_work_item}}]", "queryType": "instant", "range": false, "refId": "A" @@ -3573,9 +3573,9 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "sort_desc(max by (xrpl_network_type) (server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"}) - on(xrpl_network_type) group_right() server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"})", + "expr": "sort_desc(max by (xrpl_network_type, xrpl_work_item) (server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"}) - on(xrpl_network_type, xrpl_work_item) group_right() server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"})", "instant": true, - "legendFormat": "{{service_instance_id}} [{{xrpl_network_type}}]", + "legendFormat": "{{service_instance_id}} [{{xrpl_network_type}}] [{{xrpl_work_item}}]", "queryType": "instant", "range": false, "refId": "A" @@ -3637,9 +3637,9 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "count by (version) (build_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"})", + "expr": "count by (version, xrpl_work_item) (build_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"})", "instant": false, - "legendFormat": "{{version}}", + "legendFormat": "{{version}} [{{xrpl_work_item}}]", "queryType": "range", "range": true, "refId": "A" @@ -3988,7 +3988,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(1 / sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(ledgers_closed_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Close Interval\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(1 / sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(ledgers_closed_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Close Interval\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" } ], @@ -5002,7 +5002,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Time to fetch a missing ledger from peers, at the 95th percentile.*\n\n###### How it's computed:\n*Per-acquire durations aggregated to their 95th percentile per node over a 5-minute window.*\n\n###### Reading it:\n*Lower is better; populated mainly during sync or back-fill.*\n\n###### Healthy range:\n*Low when synced; higher and more active while catching up.*\n\n###### Watch for:\n*A spike signals the node is falling behind or recovering from a fork.*\n\n###### Keywords:\n- **Ledger acquire (inbound fetch)** *(per node)* \u2014 fetching a specific missing ledger from peers; tracked by duration and outcome (complete/failed).\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::init`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-acquire-inbound-fetch)", + "description": "###### What this is:\n*Time to fetch a missing ledger from peers, at the 95th percentile, with a separate line per acquisition outcome.*\n\n###### How it's computed:\n*Per-acquire durations aggregated to their 95th percentile per node over a 5-minute window, grouped by outcome, so complete, failed and aborted acquisitions are never mixed into one percentile.*\n\n###### Reading it:\n*Read the Complete line as the real fetch latency; lower is better, and it is populated mainly during sync or back-fill. Aborted means the acquisition was abandoned before it finished, so its span also covers the idle time it spent waiting to be reaped (at least a minute when swept, or any age when a shutdown or an admin fetch_info clear drops it) and is not comparable to a fetch time.*\n\n###### Healthy range:\n*Complete is low when synced, and higher and more active while catching up. Failed and Aborted are normally absent.*\n\n###### Watch for:\n*A spike in Complete signals the node is falling behind or recovering from a fork. A persistent Aborted line means acquisitions are routinely abandoned before they finish, so its height reflects how long they sat unfinished, not how slow the network is.*\n\n###### Keywords:\n- **Ledger acquire (inbound fetch)** *(per node)* \u2014 fetching a specific missing ledger from peers; tracked by duration and outcome \u2014 complete, failed (gave up after its retry limit, or hit unusable ledger data), or aborted (abandoned before finishing).\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::init ; InboundLedger::done ; InboundLedger::~InboundLedger`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-acquire-inbound-fetch)", "fieldConfig": { "defaults": { "color": { @@ -5093,7 +5093,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire\"}[5m]))), \"series\", \"P95 Acquire\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(histogram_quantile(0.95, sum by (le, outcome, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$outcome\", span_name=\"ledger.acquire\"}[5m]))), \"outcome\", \"Complete\", \"outcome\", \"complete\"), \"outcome\", \"Failed\", \"outcome\", \"failed\"), \"outcome\", \"Aborted\", \"outcome\", \"aborted\"), \"series\", \"P95 Acquire $1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], @@ -5196,7 +5196,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$outcome\", span_name=\"ledger.acquire\"}[$__rate_interval])), \"outcome\", \"Complete\", \"outcome\", \"complete\"), \"outcome\", \"Failed\", \"outcome\", \"failed\"), \"outcome\", \"Aborted\", \"outcome\", \"aborted\"), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], @@ -5492,6 +5492,26 @@ "refresh": 2, "sort": 1 }, + { + "name": "outcome", + "label": "Acquire Outcome", + "description": "Filter ledger acquire (inbound fetch) attempts by outcome [complete / failed / aborted]", + "type": "query", + "query": "label_values(span_calls_total{span_name=\"ledger.acquire\"}, outcome)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, { "name": "quantile", "label": "Quantile", diff --git a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json index 0617cd703a..df48858eab 100644 --- a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json +++ b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json @@ -664,19 +664,19 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_queued_us_bucket{handler=\"RcvGetObjByHash\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Queue Wait p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_queued_us_bucket{handler=\"RcvGetObjByHash\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Queue Wait p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(job_running_us_bucket{handler=\"RcvGetObjByHash\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Handler Total p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(job_running_us_bucket{handler=\"RcvGetObjByHash\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Handler Total p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_lookup_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"NodeStore Lookup p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_lookup_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"NodeStore Lookup p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -773,7 +773,7 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(sum by (result, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_lookups_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", result=~\"$result\"}[$__rate_interval])), \"series\", \"$1\", \"result\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (result, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_lookups_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", result=~\"$result\"}[$__rate_interval])), \"series\", \"$1\", \"result\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -817,7 +817,7 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_rejected_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$reason\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (reason, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_rejected_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$reason\"}[$__rate_interval])), \"series\", \"$1\", \"reason\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -861,19 +861,19 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.5, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p50\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.5, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p50\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.9, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p90\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.9, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p90\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(getobject_charge_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"Charge p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index efbb2fbca5..ca4d55239a 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -75,7 +75,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"peer.proposal.receive\"}[$__rate_interval])), \"series\", \"Proposals Received / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"peer.proposal.receive\"}[$__rate_interval])), \"series\", \"Proposals Received / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -120,7 +120,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"peer.validation.receive\"}[$__rate_interval])), \"series\", \"Validations Received / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"peer.validation.receive\"}[$__rate_interval])), \"series\", \"Validations Received / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -178,7 +178,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (proposal_trusted, service_instance_id, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", proposal_trusted=~\"$proposal_trusted\", span_name=\"peer.proposal.receive\"}[$__rate_interval])), \"series\", \"Trusted = $1\", \"proposal_trusted\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (proposal_trusted, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", proposal_trusted=~\"$proposal_trusted\", span_name=\"peer.proposal.receive\"}[$__rate_interval])), \"series\", \"Trusted = $1\", \"proposal_trusted\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -213,7 +213,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (validation_trusted, service_instance_id, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", validation_trusted=~\"$validation_trusted\", span_name=\"peer.validation.receive\"}[$__rate_interval])), \"series\", \"Trusted = $1\", \"validation_trusted\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (validation_trusted, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", validation_trusted=~\"$validation_trusted\", span_name=\"peer.validation.receive\"}[$__rate_interval])), \"series\", \"Trusted = $1\", \"validation_trusted\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json index e649350953..b751c16959 100644 --- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json +++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json @@ -113,7 +113,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Response Time\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Response Time\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -158,7 +158,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_size_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Response Size\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_size_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Response Size\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -203,21 +203,21 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.9, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P90\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.9, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P90\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_time_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -383,7 +383,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(pathfind_fast_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Fast Pathfind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(pathfind_fast_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Fast Pathfind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -428,7 +428,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(pathfind_full_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Full Pathfind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(pathfind_full_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[5m]))), \"series\", \"P95 Full Pathfind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -486,7 +486,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (method, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$grpc_method\", span_name=~\"grpc\\\\..*\"}[$__rate_interval])), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (method, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$grpc_method\", span_name=~\"grpc\\\\..*\"}[$__rate_interval])), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -531,7 +531,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, method, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$grpc_method\", span_name=~\"grpc\\\\..*\"}[5m]))), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, method, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$grpc_method\", span_name=~\"grpc\\\\..*\"}[5m]))), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -576,7 +576,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (grpc_status, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"grpc\\\\..*\", grpc_status!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"grpc_status\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (grpc_status, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"grpc\\\\..*\", grpc_status!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"grpc_status\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -621,7 +621,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.compute\"}[5m]))), \"series\", \"P95 Compute\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.compute\"}[5m]))), \"series\", \"P95 Compute\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -666,14 +666,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.request\"}[$__rate_interval])), \"series\", \"Requests / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.request\"}[$__rate_interval])), \"series\", \"Requests / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.discover\"}[$__rate_interval])), \"series\", \"Discoveries / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"pathfind.discover\"}[$__rate_interval])), \"series\", \"Discoveries / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 3ab073879d..8228c9283b 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -74,7 +74,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (command, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])) / sum by (command, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval])) * 100, \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (command, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])) / sum by (command, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval])) * 100, \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -128,7 +128,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(topk(10, sum by (command, service_instance_id, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval]))), \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(topk(10, sum by (command, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval]))), \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -166,7 +166,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.ws_message\"}[$__rate_interval])), \"series\", \"WS Messages / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.ws_message\"}[$__rate_interval])), \"series\", \"WS Messages / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -204,7 +204,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (command, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval])), \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (command, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[$__rate_interval])), \"series\", \"$1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -249,7 +249,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, command, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[5m]))), \"series\", \"P95 $1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, command, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\"}[5m]))), \"series\", \"P95 $1\", \"command\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -343,14 +343,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.http_request\"}[$__rate_interval])), \"series\", \"rpc.http_request / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.http_request\"}[$__rate_interval])), \"series\", \"rpc.http_request / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.process\"}[$__rate_interval])), \"series\", \"rpc.process / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=\"rpc.process\"}[$__rate_interval])), \"series\", \"rpc.process / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -395,14 +395,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_OK\"}[$__rate_interval])), \"series\", \"Success\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_OK\"}[$__rate_interval])), \"series\", \"Success\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])), \"series\", \"Error\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", command=~\"$command\", span_name=~\"rpc.command.*\", status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])), \"series\", \"Error\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -452,7 +452,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, load_type, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"rpc.command.*\", load_type!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"load_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, load_type, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"rpc.command.*\", load_type!=\"\"}[$__rate_interval])), \"series\", \"$1\", \"load_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -497,14 +497,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"rpc.process\", is_batch=\"true\"}[$__rate_interval])), \"series\", \"Batch\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"rpc.process\", is_batch=\"true\"}[$__rate_interval])), \"series\", \"Batch\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"rpc.process\", is_batch=\"false\"}[$__rate_interval])), \"series\", \"Single\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"rpc.process\", is_batch=\"false\"}[$__rate_interval])), \"series\", \"Single\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -566,7 +566,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (service_instance_id, le, xrpl_branch, xrpl_node_role) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p99 Latency\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (service_instance_id, le, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p99 Latency\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -622,21 +622,21 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_started_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Started/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_started_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Started/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_finished_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Finished/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_finished_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Finished/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_errored_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Errored/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_errored_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[$__rate_interval])), \"series\", \"Errored/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -793,14 +793,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.75, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p75\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.75, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p75\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m]))), \"series\", \"p99\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -854,7 +854,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(topk(10, histogram_quantile(0.99, sum by (le, method, service_instance_id, xrpl_branch, xrpl_node_role) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m])))), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(topk(10, histogram_quantile(0.99, sum by (le, method, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(rpc_method_us_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", method=~\"$method\"}[5m])))), \"series\", \"$1\", \"method\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/transaction-overview.json b/docker/telemetry/grafana/dashboards/transaction-overview.json index a7d2ad429a..f5e6820a81 100644 --- a/docker/telemetry/grafana/dashboards/transaction-overview.json +++ b/docker/telemetry/grafana/dashboards/transaction-overview.json @@ -61,7 +61,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.transactor\", stage=\"apply\", ter_result!~\"tesSUCCESS|\"}[$__rate_interval])), \"series\", \"Failed / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.transactor\", stage=\"apply\", ter_result!~\"tesSUCCESS|\"}[$__rate_interval])), \"series\", \"Failed / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -120,7 +120,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, tx_type, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[5m]))), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, tx_type, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[5m]))), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -183,7 +183,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (tx_type, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (tx_type, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -233,7 +233,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (tx_type, ter_result, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\", ter_result=~\"$ter_result\", ter_result!=\"tesSUCCESS\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"ter_result\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (tx_type, ter_result, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\", ter_result=~\"$ter_result\", ter_result!=\"tesSUCCESS\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"ter_result\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -278,7 +278,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (suppressed, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{span_name=\"tx.receive\", tx_type=~\"$tx_type\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Suppressed $1\", \"suppressed\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (suppressed, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{span_name=\"tx.receive\", tx_type=~\"$tx_type\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Suppressed $1\", \"suppressed\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -323,14 +323,14 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"tx.process / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"tx.process / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.receive\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"tx.receive / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.receive\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"tx.receive / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -382,7 +382,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (local, service_instance_id, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"Local $1\", \"local\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (local, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"Local $1\", \"local\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "id": 8 @@ -464,7 +464,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, tx_type, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.transactor\", tx_type=~\"$tx_type\"}[5m]))), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, tx_type, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.transactor\", tx_type=~\"$tx_type\"}[5m]))), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -518,9 +518,9 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (service_instance_id, xrpl_network_type) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept_tx\", txq_status=\"applied\"}[$__rate_interval]))\n/\nsum by (service_instance_id, xrpl_network_type) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept_tx\", txq_status=~\"applied|failed\"}[$__rate_interval]))", + "expr": "sum by (service_instance_id, xrpl_network_type, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept_tx\", txq_status=\"applied\"}[$__rate_interval]))\n/\nsum by (service_instance_id, xrpl_network_type, xrpl_work_item) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept_tx\", txq_status=~\"applied|failed\"}[$__rate_interval]))", "interval": "15s", - "legendFormat": "{{service_instance_id}} [{{xrpl_network_type}}]" + "legendFormat": "{{service_instance_id}} [{{xrpl_network_type}}] [{{xrpl_work_item}}]" } ], "fieldConfig": { @@ -560,7 +560,7 @@ }, { "title": "Tx Apply Pipeline Rate by Stage", - "description": "**What:** Throughput of each apply-pipeline stage (preflight, preclaim, apply), showing where transactions drop out.\n**How it's computed:** Per-second rate per stage over a 5-minute window, per node.\n**Reading it:** A decline from earlier to later stages shows where transactions are filtered.\n**Healthy range:** Workload-dependent; later stages sit at or below earlier ones.\n**Watch for:** A large early-stage drop means many transactions fail basic checks, consistent with malformed floods.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", + "description": "###### What this is:\n*Throughput of each apply-pipeline stage (preflight, preclaim, apply), showing where transactions drop out.*\n\n###### How it's computed:\n*Per-second rate of stage spans over the rate interval, grouped by stage and node.*\n\n###### Reading it:\n*A decline from earlier to later stages shows where transactions are filtered.*\n\n###### Healthy range:\n*Workload-dependent; later stages sit at or below earlier ones.*\n\n###### Watch for:\n*A large early-stage drop, meaning many transactions fail basic checks, consistent with malformed floods.*\n\n###### Keywords:\n- **Apply pipeline stages** *(per node)* \u2014 the ordered checks a transaction passes \u2014 preflight (stateless), preclaim (stateful), then apply.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[applySteps.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/applySteps.cpp) \u00b7 [Transactor.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/Transactor.cpp)\n\n###### Function:\n`makeStageSpan ; Transactor::operator()`\n\n###### References:\n[Apply pipeline stages](https://xrpl.org/docs/concepts/transactions) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#apply-pipeline-stages)", "type": "timeseries", "gridPos": { "h": 10, @@ -583,15 +583,16 @@ "targets": [ { "datasource": { - "type": "prometheus" + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" }, - "expr": "sum by (stage, service_instance_id) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\"}[$__rate_interval]))", - "interval": "15s", - "legendFormat": "{{stage}} [{{service_instance_id}}]" + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\"}[$__rate_interval])), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "interval": "15s" } ], "fieldConfig": { "defaults": { + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", "unit": "ops", "custom": { "axisLabel": "Spans / Sec", @@ -649,7 +650,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(histogram_quantile(0.95, sum by (le, stage, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\"}[5m]))), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(histogram_quantile(0.95, sum by (le, stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\"}[5m]))), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -699,7 +700,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\", ter_result!~\"tesSUCCESS|\"}[$__rate_interval])), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", stage=~\"$stage\", ter_result!~\"tesSUCCESS|\"}[$__rate_interval])), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -749,7 +750,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(histogram_quantile(0.95, sum by (le, tx_type, stage, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", tx_type=~\"$tx_type\", stage=~\"$stage\"}[5m]))), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"stage\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(histogram_quantile(0.95, sum by (le, tx_type, stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", tx_type=~\"$tx_type\", stage=~\"$stage\"}[5m]))), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"stage\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -771,6 +772,106 @@ }, "id": 16 }, + { + "title": "Tx Apply Pipeline Rate by Type and Stage", + "description": "###### What this is:\n*Throughput of each apply-pipeline stage broken down by both transaction type and pipeline stage.*\n\n###### How it's computed:\n*Per-second rate of stage spans over the rate interval, grouped by transaction type, stage and node; higher cardinality than the by-stage view.*\n\n###### Reading it:\n*A decline from earlier to later stages shows where each transaction type is filtered out.*\n\n###### Healthy range:\n*Workload-dependent; for every type the later stages sit at or below the earlier ones.*\n\n###### Watch for:\n*One type's early-stage rate dwarfing its later stages, consistent with a malformed flood of that type.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n- **Apply pipeline stages** *(per node)* \u2014 the ordered checks a transaction passes \u2014 preflight (stateless), preclaim (stateful), then apply.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[applySteps.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/applySteps.cpp) \u00b7 [Transactor.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/Transactor.cpp)\n\n###### Function:\n`makeStageSpan ; Transactor::operator()`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Apply pipeline stages](https://xrpl.org/docs/concepts/transactions) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", + "type": "timeseries", + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 122 + }, + "options": { + "tooltip": { + "mode": "multi", + "sort": "desc", + "maxHeight": 600 + }, + "legend": { + "displayMode": "table", + "placement": "right", + "calcs": ["mean", "max"] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (tx_type, stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", tx_type=~\"$tx_type\", stage=~\"$stage\"}[$__rate_interval])), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"stage\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + } + ], + "fieldConfig": { + "defaults": { + "unit": "ops", + "custom": { + "axisLabel": "Spans / Sec", + "spanNulls": 1800000, + "insertNulls": false, + "showPoints": "auto", + "pointSize": 5, + "lineWidth": 1, + "fillOpacity": 0, + "gradientMode": "none" + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}" + }, + "overrides": [] + }, + "id": 22 + }, + { + "title": "Tx Apply Pipeline Failure Rate by Type and Stage", + "description": "###### What this is:\n*How many transactions fail at each apply-pipeline stage per second, broken down by transaction type.*\n\n###### How it's computed:\n*Per-second rate of non-success outcomes over the rate interval, grouped by transaction type, stage and node; higher cardinality than the by-stage view.*\n\n###### Reading it:\n*Shows which transaction type is being rejected and at which stage \u2014 preflight, preclaim, or apply.*\n\n###### Healthy range:\n*Workload-dependent; a modest background of expected rejections is normal for every type.*\n\n###### Watch for:\n*A failure spike concentrated in one type and stage pair, consistent with malformed or spam submissions of that type.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n- **Apply pipeline stages** *(per node)* \u2014 the ordered checks a transaction passes \u2014 preflight (stateless), preclaim (stateful), then apply.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[applySteps.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/applySteps.cpp) \u00b7 [Transactor.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/tx/Transactor.cpp)\n\n###### Function:\n`makeStageSpan ; Transactor::operator()`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Apply pipeline stages](https://xrpl.org/docs/concepts/transactions) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", + "type": "timeseries", + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 132 + }, + "options": { + "tooltip": { + "mode": "multi", + "sort": "desc", + "maxHeight": 600 + }, + "legend": { + "displayMode": "table", + "placement": "right", + "calcs": ["mean", "max"] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(label_replace(label_replace(label_replace(sum by (tx_type, stage, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=~\"tx.preflight|tx.preclaim|tx.transactor\", tx_type=~\"$tx_type\", stage=~\"$stage\", ter_result!~\"tesSUCCESS|\"}[$__rate_interval])), \"stage\", \"Preflight\", \"stage\", \"preflight\"), \"stage\", \"Preclaim\", \"stage\", \"preclaim\"), \"stage\", \"Apply\", \"stage\", \"apply\"), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"stage\", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + } + ], + "fieldConfig": { + "defaults": { + "unit": "suffix: failures/s", + "custom": { + "axisLabel": "Failures / Sec", + "spanNulls": 1800000, + "insertNulls": false, + "showPoints": "auto", + "pointSize": 5, + "lineWidth": 1, + "fillOpacity": 0, + "gradientMode": "none" + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}" + }, + "overrides": [] + }, + "id": 23 + }, { "title": "Transaction Apply Duration per Ledger", "description": "###### What this is:\n*The 95th-percentile time to apply the agreed transaction set into each new ledger.*\n\n###### How it's computed:\n*95th-percentile of transaction-apply durations over 5 minutes, per node.*\n\n###### Reading it:\n*Lower is better; a major component of ledger build time.*\n\n###### Healthy range:\n*A few to tens of milliseconds; scales with transactions per ledger.*\n\n###### Watch for:\n*Rising durations during heavy or expensive transaction sets.*\n\n###### Keywords:\n- **Transaction apply phase** *(per node)* \u2014 the step that executes the agreed transaction set into the new ledger during a close.\n- **Ledger build** *(per node)* \u2014 constructing the new ledger by applying the agreed transaction set to the prior ledger.\n- **In-ledger vs target count** *(per node)* \u2014 transactions already in the open ledger versus the soft target that triggers fee escalation.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in code as a trace span, turned into a metric by the collector (SpanMetrics connector), then aggregated by the Grafana query.*\n\n###### Source:\n[BuildLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/BuildLedger.cpp)\n\n###### Function:\n`applyTransactions`\n\n###### References:\n[Transaction apply phase](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [In-ledger vs target count](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-apply-phase)", @@ -779,7 +880,7 @@ "h": 10, "w": 12, "x": 0, - "y": 122 + "y": 142 }, "options": { "tooltip": { @@ -794,7 +895,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[5m]))), \"series\", \"tx.apply\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"tx.apply\"}[5m]))), \"series\", \"tx.apply\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -823,7 +924,7 @@ "h": 1, "w": 24, "x": 0, - "y": 132 + "y": 152 }, "collapsed": false, "panels": [], @@ -837,7 +938,7 @@ "h": 10, "w": 12, "x": 0, - "y": 133 + "y": 153 }, "options": { "tooltip": { @@ -852,7 +953,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (tx_type, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.enqueue\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (tx_type, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.enqueue\"}[$__rate_interval])), \"series\", \"$1\", \"tx_type\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -882,7 +983,7 @@ "h": 10, "w": 12, "x": 12, - "y": 133 + "y": 153 }, "options": { "tooltip": { @@ -897,7 +998,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept\"}[5m]))), \"series\", \"Drain\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.accept\"}[5m]))), \"series\", \"Drain\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { @@ -927,7 +1028,7 @@ "h": 10, "w": 12, "x": 0, - "y": 143 + "y": 163 }, "options": { "tooltip": { @@ -942,7 +1043,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.cleanup\"}[$__rate_interval])), \"series\", \"Cleanups / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"txq.cleanup\"}[$__rate_interval])), \"series\", \"Cleanups / Sec\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/dashboards/validate_dashboards.py b/docker/telemetry/grafana/dashboards/validate_dashboards.py index d1b17dbbf6..27b8d36469 100755 --- a/docker/telemetry/grafana/dashboards/validate_dashboards.py +++ b/docker/telemetry/grafana/dashboards/validate_dashboards.py @@ -2,6 +2,7 @@ """Dashboard lint: cumulative metrics rate()-wrapped; tier filters; sane panel grid.""" import json, re, sys +from pathlib import Path GRID_COLUMNS = 24 @@ -39,6 +40,7 @@ NODESTORE_CUMULATIVE = ( "node_read_bytes", "node_written_bytes", "node_reads_duration_us", + "node_writes_duration_us", ) # state_accounting_*_duration are cumulative µs. STATE_DURATION = re.compile(r"state_accounting_\w+_duration") @@ -163,8 +165,23 @@ def check(path, forbid_5m): def main(): + """Validate the dashboards named on the command line, or all of them. + + Exit status: 0 every dashboard passed, 1 violations were found, 2 there was + nothing to validate. The last is an error rather than a pass: reporting + success without having read a single dashboard is indistinguishable from a + clean run, so a caller that mis-spells a path gets a green light for work + that never happened. + """ args = [a for a in sys.argv[1:] if not a.startswith("--")] forbid_5m = "--no-5m" in sys.argv + if not args: + # Default to every dashboard beside this script, so a bare run checks + # the whole set instead of iterating an empty list. + args = sorted(str(p) for p in Path(__file__).resolve().parent.glob("*.json")) + if not args: + print("ERROR: no dashboard JSON files to validate", file=sys.stderr) + sys.exit(2) all_errs = [] for path in args: all_errs += check(path, forbid_5m) diff --git a/docker/telemetry/grafana/dashboards/validator-health.json b/docker/telemetry/grafana/dashboards/validator-health.json index df76ace0b3..15254cb9ca 100644 --- a/docker/telemetry/grafana/dashboards/validator-health.json +++ b/docker/telemetry/grafana/dashboards/validator-health.json @@ -801,7 +801,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(3600 * sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(state_changes_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Changes/hr\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" + "expr": "label_replace(label_join(label_replace(3600 * sum by (service_instance_id, xrpl_branch, xrpl_node_role, xrpl_work_item) (rate(state_changes_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Changes/hr\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")" } ], "fieldConfig": { diff --git a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml index 03fa2344b4..06407f904e 100644 --- a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml @@ -1,6 +1,6 @@ # Grafana contact-point provisioning for rippled OTel alerts. # -# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# Alerting on health-critical internal metrics. # # A contact point is where a firing alert is delivered. Two are defined: # xrpld-default — Slack only; receives warning-severity alerts. @@ -41,10 +41,11 @@ # GF_SMTP_ENABLED=true and the relay settings point somewhere real. # # Grafana Cloud does NOT use this file — Cloud has no provisioning filesystem. -# Cloud delivery is created over the REST API by upload_alerts_to_grafana.py, -# which builds a single email-only contact point and attaches it to each rule -# via per-rule notification_settings. See that script's header for why the -# notification policy tree must not be touched on a shared Cloud instance. +# Cloud delivery is created over the REST API instead: a single email-only +# contact point, attached to each rule via per-rule notification_settings. +# The notification policy tree must never be pushed to a shared Cloud +# instance -- there is exactly one tree per org and the PUT endpoint +# replaces it wholesale. apiVersion: 1 diff --git a/docker/telemetry/grafana/provisioning/alerting/policies.yaml b/docker/telemetry/grafana/provisioning/alerting/policies.yaml index b1c4c92c70..3b65bb4ae6 100644 --- a/docker/telemetry/grafana/provisioning/alerting/policies.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/policies.yaml @@ -1,6 +1,6 @@ # Grafana notification-policy provisioning for rippled OTel alerts. # -# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# Alerting on health-critical internal metrics. # # The notification policy tree decides which contact point receives a firing # alert and how alerts are batched. Routing is split by severity: diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml index a66e1fcceb..9746559b28 100644 --- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -1,8 +1,8 @@ # Grafana alert-rule provisioning for rippled OTel metrics. # -# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# Alerting on health-critical internal metrics. # -# Twelve rules across five subsystems: consensus/ledger health, validator +# Thirteen rules across five subsystems: consensus/ledger health, validator # health, the job queue, node operating state, and the overlay (manifests). # # Rule shape (Grafana server-side evaluation): @@ -845,10 +845,10 @@ groups: # 24h window that an earlier revision used: # healthy p95 0.2-0.5 kB/s, p99 1.0-1.8 kB/s # observed peaks up to 2.7 MB/s during real manifest storms - # 512 kB/s sits ~280x above healthy p99 and ~5x below the peaks. An - # earlier 50 kB/s threshold produced ~41 sustained 5-min samples across - # six healthy nodes over six days (i.e. routine paging); 512 kB/s reduces - # that to 2 while still catching every genuine storm. + # 512 KiB/s (524288 B/s) sits ~280x above healthy p99 and ~5x below the + # peaks. An earlier 50 kB/s threshold produced ~41 sustained 5-min samples + # across six healthy nodes over six days (i.e. routine paging); + # 512 KiB/s reduces that to 2 while still catching every genuine storm. # # The uptime gate exists because the startup manifest burst is MEASURED # NORMAL behaviour. It does not hide real floods — the same 7-day sample @@ -869,7 +869,8 @@ groups: summary: "Inbound manifest flood on {{ $labels.service_instance_id }}" description: >- Node {{ $labels.service_instance_id }} is receiving - {{ $values.B.Value }} B/s of manifest traffic (>512 kB/s) over 10m. + {{ $values.B.Value }} B/s of manifest traffic over 10m, above the + 512 KiB/s (524288 B/s) threshold. A peer is flooding oversized TMManifests dumps. data: - refId: A diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index b4401d5906..2254ac4399 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -62,7 +62,10 @@ die() { check_span() { local op="$1" local count - count=$(curl -sf "$TEMPO/api/search" \ + # -G is required: it moves the urlencoded params into the query string. + # Without it curl POSTs them as a request body, and Tempo answers 200 + # while ignoring the query — so every span name would look present. + count=$(curl -sfG "$TEMPO/api/search" \ --data-urlencode "q={resource.service.name=\"xrpld\" && name=\"$op\"}" \ --data-urlencode "limit=5" | jq '.traces | length' 2>/dev/null || echo 0) @@ -184,6 +187,23 @@ mkdir -p "$WORKDIR" # --------------------------------------------------------------------------- # Step 2: Start observability stack # --------------------------------------------------------------------------- + +# From here on the script owns the docker stack and the xrpld nodes, so an +# abort must tear them down instead of leaving them behind. A run that +# reaches the summary deliberately leaves everything up for inspection +# (see the header comment), so the trap only fires before that point. +RUN_COMPLETED=0 +on_exit() { + local status=$? + if [ "$RUN_COMPLETED" -eq 0 ]; then + log "Aborted with exit status $status — tearing down." + cleanup + fi +} +trap on_exit EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + log "Starting observability stack..." # Point the collector's log mount at this test's workdir so it tails the # per-node debug.log files this script generates. The compose default @@ -382,11 +402,6 @@ endpoint=http://localhost:4318/v1/metrics prefix=rippled service_instance_id=Node-${i} -[insight] -server=statsd -address=127.0.0.1:8125 -prefix=rippled - [rpc_startup] { "command": "log_level", "severity": "warning" } @@ -419,12 +434,14 @@ log "Waiting for nodes to reach 'proposing' state (timeout: ${CONSENSUS_TIMEOUT} start_time=$(date +%s) nodes_ready=0 +consensus_timed_out=0 while [ "$nodes_ready" -lt "$NUM_NODES" ]; do elapsed=$(($(date +%s) - start_time)) if [ "$elapsed" -ge "$CONSENSUS_TIMEOUT" ]; then fail "Consensus timeout after ${CONSENSUS_TIMEOUT}s ($nodes_ready/$NUM_NODES nodes ready)" log "Continuing with partial consensus..." + consensus_timed_out=1 break fi @@ -447,7 +464,10 @@ echo "" if [ "$nodes_ready" -eq "$NUM_NODES" ]; then ok "All $NUM_NODES nodes reached 'proposing' state" -else +elif [ "$consensus_timed_out" -eq 0 ]; then + # The timeout branch above already called fail(), so reporting again here + # would count one timeout twice. Only reachable if the loop ever gains + # another early exit. fail "Only $nodes_ready/$NUM_NODES nodes reached 'proposing' state" fi @@ -470,7 +490,7 @@ for attempt in $(seq 1 60); do done # --------------------------------------------------------------------------- -# Step 7: Exercise RPC spans (Phase 2) +# Step 7: Exercise RPC spans # --------------------------------------------------------------------------- log "Exercising RPC spans..." @@ -485,15 +505,17 @@ log "RPC commands sent. Waiting 5s for batch export..." sleep 5 # --------------------------------------------------------------------------- -# Step 8: Submit transaction (Phase 3) +# Step 8: Submit transaction # --------------------------------------------------------------------------- log "Submitting Payment transaction..." # Generate a destination wallet log " Generating destination wallet..." +# Guarded: under set -e an unguarded curl failure would abort the whole +# script, so the fallback below could never run. wallet_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d '{"method":"wallet_propose"}') -DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null) + -d '{"method":"wallet_propose"}') || wallet_result="" +DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null || echo "") if [ -z "$DEST_ACCOUNT" ] || [ "$DEST_ACCOUNT" = "null" ]; then fail "Could not generate destination wallet" DEST_ACCOUNT="rrrrrrrrrrrrrrrrrrrrrhoLvTp" # ACCOUNT_ZERO fallback @@ -502,13 +524,13 @@ log " Destination: $DEST_ACCOUNT" # Get genesis account info acct_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}") + -d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}") || acct_result="" seq_num=$(echo "$acct_result" | jq -r '.result.account_data.Sequence' 2>/dev/null || echo "unknown") log " Genesis account sequence: $seq_num" # Submit payment submit_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}") + -d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}") || submit_result="" engine_result=$(echo "$submit_result" | jq -r '.result.engine_result' 2>/dev/null || echo "unknown") tx_hash=$(echo "$submit_result" | jq -r '.result.tx_json.hash' 2>/dev/null || echo "unknown") @@ -538,39 +560,39 @@ else fi log "" -log "--- Phase 2: RPC Spans ---" -check_span "rpc.request" +log "--- RPC Spans ---" +check_span "rpc.http_request" check_span "rpc.process" check_span "rpc.command.server_info" check_span "rpc.command.server_state" check_span "rpc.command.ledger" log "" -log "--- Phase 3: Transaction Spans ---" +log "--- Transaction Spans ---" check_span "tx.process" check_span "tx.receive" check_span "tx.apply" log "" -log "--- Phase 4: Consensus Spans ---" +log "--- Consensus Spans ---" check_span "consensus.proposal.send" check_span "consensus.ledger_close" check_span "consensus.accept" check_span "consensus.validation.send" log "" -log "--- Phase 5: Ledger Spans ---" +log "--- Ledger Spans ---" check_span "ledger.build" check_span "ledger.validate" check_span "ledger.store" log "" -log "--- Phase 5: Peer Spans (trace_peer=1) ---" +log "--- Peer Spans (trace_peer=1) ---" check_span "peer.proposal.receive" check_span "peer.validation.receive" # --------------------------------------------------------------------------- -# Step 9b: Verify log-trace correlation (Phase 8) +# Step 9b: Verify log-trace correlation # --------------------------------------------------------------------------- log "" log "--- Log-Trace Correlation ---" @@ -580,7 +602,7 @@ check_log_correlation # Step 10: Verify Prometheus spanmetrics # --------------------------------------------------------------------------- log "" -log "--- Phase 5: Spanmetrics ---" +log "--- Spanmetrics ---" log "Waiting 20s for Prometheus scrape cycle..." sleep 20 @@ -611,7 +633,7 @@ fi # Step 10b: Verify native OTel metrics in Prometheus (beast::insight) # --------------------------------------------------------------------------- log "" -log "--- Phase 7: Native OTel Metrics (beast::insight via OTLP) ---" +log "--- Native OTel Metrics (beast::insight via OTLP) ---" log "Waiting 20s for OTLP metric export + Prometheus scrape..." sleep 20 @@ -630,7 +652,7 @@ check_otel_metric() { # Node health gauges (ObservableGauge — no _total suffix) check_otel_metric "ledgermaster_validated_ledger_age" check_otel_metric "ledgermaster_published_ledger_age" -check_otel_metric "job_count" +check_otel_metric "jobq_job_count" # State accounting check_otel_metric "state_accounting_full_duration" @@ -660,10 +682,10 @@ else fi # --------------------------------------------------------------------------- -# Step 10c: Verify Phase 9 OTel SDK Metrics +# Step 10c: Verify OTel SDK Metrics # --------------------------------------------------------------------------- log "" -log "--- Phase 9: OTel SDK Metrics (MetricsRegistry) ---" +log "--- OTel SDK Metrics (MetricsRegistry) ---" log "Waiting 15s for OTel metric export + Prometheus scrape..." sleep 15 @@ -679,34 +701,34 @@ check_otel_metric() { fi } -# Task 9.1: NodeStore I/O +# NodeStore I/O check_otel_metric 'nodestore_state{metric="node_reads_total"}' check_otel_metric 'nodestore_state{metric="write_load"}' -# Task 9.2: Cache hit rates +# Cache hit rates check_otel_metric 'cache_metrics{metric="SLE_hit_rate"}' check_otel_metric 'cache_metrics{metric="treenode_cache_size"}' -# Task 9.3: TxQ metrics +# TxQ metrics check_otel_metric 'txq_metrics{metric="txq_count"}' check_otel_metric 'txq_metrics{metric="txq_reference_fee_level"}' -# Task 9.4: Per-RPC metrics +# Per-RPC metrics check_otel_metric "rpc_method_started_total" check_otel_metric "rpc_method_finished_total" -# Task 9.5: Per-job metrics +# Per-job metrics check_otel_metric "job_queued_total" check_otel_metric "job_finished_total" -# Task 9.6: Counted object instances +# Counted object instances check_otel_metric "object_count" -# Task 9.7: Load factor breakdown +# Load factor breakdown check_otel_metric 'load_factor_metrics{metric="load_factor"}' check_otel_metric 'load_factor_metrics{metric="load_factor_server"}' -# Task 7.15 / Phase 9: ValidationTracker rolling-window agreement gauge. +# ValidationTracker rolling-window agreement gauge. # MetricsRegistry::registerValidationAgreementGauge() publishes # validation_agreement with a `metric` label for each window # (1h / 24h / 7d) plus the matching agreement/miss counts. The 7-day @@ -724,6 +746,11 @@ check_otel_metric 'validation_agreement{metric="missed_7d"}' # --------------------------------------------------------------------------- # Step 11: Summary # --------------------------------------------------------------------------- + +# All checks are done, so the run counts as complete: keep the stack and the +# nodes up for inspection even when some checks failed. +RUN_COMPLETED=1 + echo "" echo "===========================================================" echo " INTEGRATION TEST RESULTS" diff --git a/docker/telemetry/otel-collector-config.yaml b/docker/telemetry/otel-collector-config.yaml index 1a8cde20ec..c6bc3a2804 100644 --- a/docker/telemetry/otel-collector-config.yaml +++ b/docker/telemetry/otel-collector-config.yaml @@ -3,7 +3,7 @@ # Pipelines: # traces: OTLP receiver -> batch processor -> debug + Tempo + spanmetrics # metrics: OTLP receiver + spanmetrics connector -> Prometheus exporter -# logs: filelog receiver -> batch processor -> otlphttp/Loki (Phase 8) +# logs: filelog receiver -> batch processor -> otlphttp/Loki # # xrpld sends traces via OTLP/HTTP to port 4318. The collector batches # them, forwards to Tempo, and derives RED metrics via the spanmetrics @@ -37,6 +37,16 @@ receivers: # optional — only present when the log was emitted within an active span. filelog: include: [/var/log/xrpld/*/debug.log] + # Read each file from the start. The upstream default is `end`, which + # skips everything written before the receiver's first poll — so any log + # line a node emitted before the collector got to it would be lost, and + # nothing is read at all from a file that has stopped being written to. + # + # Offsets are kept in memory here, so a restarted collector re-reads the + # file. Stacks that keep their logs across restarts layer + # otel-collector-filestorage.yaml on top to persist them; ephemeral + # stacks get a fresh log directory each run and need nothing. + start_at: beginning operators: # Log format emitted by Logs::format() is: # YYYY-Mmm-DD HH:MM:SS.ffffff UTC : [trace_id=... span_id=...] @@ -56,18 +66,24 @@ processors: send_batch_size: 100 resource/logs: attributes: + # Loki 3.x OTLP ingestion promotes only its own allow-list of resource + # attributes to stream (index) labels; `service.name` is on that list + # and arrives as the label `service_name`, which is what the LogQL + # examples in the runbook and TESTING.md select on. + # + # A custom `job` attribute is NOT on that list. Verified against + # grafana/loki:3.4.2 with the default config: after ingesting through + # this pipeline, /loki/api/v1/labels returned only `service_name` and + # `deployment_environment`, `{job="xrpld"}` matched 0 streams, and + # `job` appeared as structured metadata instead — which a `{...}` + # stream selector cannot match. Promoting it would mean mounting a Loki + # config and adding it to limits_config.otlp_config.resource_attributes + # (additive to Loki's defaults unless ignore_defaults is set), which is + # not worth a constant value — especially as Loki caps index labels at + # 15 and already promotes ~17 by default. Select on `service_name`. - key: service.name value: xrpld action: upsert - # Loki 3.x OTLP ingestion converts `service.name` to the label - # `service_name`. The runbook and integration-test queries use the - # canonical Loki label `job` so operators can paste `{job="xrpld"}` - # without guessing the otel-to-loki naming convention. Upsert the - # `job` resource attribute here so it round-trips through OTLP - # into Loki as the `job` label. - - key: job - value: xrpld - action: upsert # Deployment-tier tagging. Each collector serves ONE environment and ONE # network, so it stamps both onto every signal it forwards. This lets a # single Grafana stack hold data from many collectors and filter by tier. diff --git a/docker/telemetry/otel-collector-filestorage.yaml b/docker/telemetry/otel-collector-filestorage.yaml new file mode 100644 index 0000000000..5362431725 --- /dev/null +++ b/docker/telemetry/otel-collector-filestorage.yaml @@ -0,0 +1,28 @@ +# Collector overlay that persists filelog read offsets. Applied ONLY by the +# developer stack (docker/telemetry/docker-compose.yml), as a second --config +# after otel-collector-config.yaml; the collector deep-merges the two. +# +# Why this is an overlay rather than part of the base config: the base config +# is shared by every stack that runs the collector, including the ephemeral +# workload-validation stack, which creates a fresh log directory per run and so +# has nothing to resume from. The extension needs a writable directory, and the +# collector image runs as 10001:10001 with no writable path of its own, so +# requiring it in the base config would force every stack to mount a volume +# just to start. Keeping it here means the base config stays self-sufficient. +# +# The developer stack benefits because its log directory and this volume both +# survive `docker compose down`, so a restart resumes at the last offset +# instead of re-reading debug.log from the top. + +extensions: + file_storage/filelog: + directory: /var/lib/otelcol/file_storage + create_directory: true + +receivers: + filelog: + storage: file_storage/filelog + +# Lists are replaced rather than merged, so this must repeat the base entry. +service: + extensions: [health_check, file_storage/filelog] diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md index 147ac79be0..e34608192a 100644 --- a/docker/telemetry/workload/README.md +++ b/docker/telemetry/workload/README.md @@ -5,10 +5,14 @@ Synthetic workload generation and validation tools for xrpld's OpenTelemetry tel ## Quick Start ```bash -# Build xrpld with telemetry enabled -conan install . --build=missing -o telemetry=True -cmake --preset default -Dtelemetry=ON -cmake --build --preset default +# Build xrpld with telemetry enabled (see BUILD.md for the full flow) +mkdir -p .build && cd .build +conan install .. --output-folder . --build missing \ + --settings build_type=Release -o telemetry=True +cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \ + -DCMAKE_BUILD_TYPE=Release -Dtelemetry=ON .. +cmake --build . --parallel "$(nproc)" --target xrpld +cd .. # Run full validation (starts everything, runs load, validates) docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld @@ -27,16 +31,19 @@ spans (proposals, validations), and all metric pipelines. run-full-validation.sh (shell orchestrator) | |-- docker-compose.workload.yaml - | |-- otel-collector (traces via OTLP + StatsD receiver) + | |-- otel-collector (otlp receiver: traces + beast::insight metrics; + | | filelog receiver: node debug.log -> Loki) | |-- tempo (trace backend + TraceQL search API) | |-- prometheus (metrics scraping) + | |-- loki (log aggregation for log-trace correlation) | |-- grafana (dashboards, provisioned automatically) | |-- generate-validator-keys.sh | -> validator-keys.json, validators.txt | |-- Nx xrpld nodes (local processes, full telemetry) - | - Each node: [telemetry] enabled=1, trace_rpc/consensus/transactions + | - Each node: [telemetry] enabled=1, all 5 trace_* categories on + | - [insight] server=otel (beast::insight metrics over OTLP, no StatsD) | - [signing_support] true (server-side signing for tx_submitter) | - Peer discovery via [ips] (not [ips_fixed]) for active peer counts | @@ -49,6 +56,7 @@ run-full-validation.sh (shell orchestrator) | -> validation-report.json | |-- benchmark.sh (baseline vs telemetry comparison) + |-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling) -> benchmark-report-*.md ``` @@ -60,22 +68,27 @@ each phase, the RPC generator and TX submitter run concurrently. ### Available Profiles -| Profile | Phases | Duration | Purpose | -| ----------------- | ------ | ---------------------------- | ----------------------------------------------------------- | -| `full-validation` | 6 | ~5 min + 1 min propagation | Full 18-dashboard coverage with burst/idle/plateau patterns | -| `quick-smoke` | 1 | ~30s + 30s propagation | Fast CI smoke test | -| `stress` | 3 | ~3.5 min + 1 min propagation | Heavy sustained load for benchmarking | +| Profile | Phases | Duration | Purpose | +| ----------------- | ------ | --------------------------- | ------------------------------------------------------------------------------------------------ | +| `full-validation` | 7 | 4.5 min + 1 min propagation | Coverage for the full asserted span/metric/dashboard inventory, with burst/idle/plateau patterns | +| `quick-smoke` | 1 | 30s + 30s propagation | Fast CI smoke test | +| `stress` | 3 | 3.5 min + 1 min propagation | Heavy sustained load for benchmarking | + +Durations are the sum of the phase `duration_sec` values in +`workload-profiles.json` plus that profile's `propagation_wait_sec`; they exclude +cluster startup and the validation pass itself. ### full-validation Phases -| Phase | RPC Rate | TX TPS | Duration | Dashboard Coverage | -| ------------ | -------- | ------ | -------- | ----------------------------------------------- | -| warmup | 5 RPS | — | 30s | Node Health, Validator Health (baseline gauges) | -| steady-state | 30 RPS | 3 TPS | 60s | All dashboards (plateau data) | -| rpc-burst | 100 RPS | — | 30s | Job Queue, RPC Performance (latency spikes) | -| tx-flood | 5 RPS | 20 TPS | 30s | Fee Market & TxQ, Transaction Overview | -| mixed-peak | 50 RPS | 10 TPS | 60s | Consensus Health, Ledger Operations | -| cooldown | 5 RPS | — | 30s | Recovery patterns, state transitions | +| Phase | RPC Rate | TX TPS | Duration | Dashboard Coverage | +| ------------ | ------------------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| warmup | 5 RPS | — | 30s | Node Health, Validator Health (baseline gauges) | +| steady-state | 30 RPS | 3 TPS | 60s | All dashboards (plateau data) | +| rpc-burst | 100 RPS | — | 30s | Job Queue, RPC Performance (latency spikes) | +| tx-flood | 5 RPS | 20 TPS | 30s | Fee Market & TxQ, Transaction Overview | +| txq-burst | 5 RPS (100% `fee`) | 60 TPS | 30s | Fee Market & TxQ — single-type Payment burst that forces open-ledger fee escalation and TxQ queueing, exercising the `txq.*` spans (`txq.enqueue`, `txq.accept`, `txq.accept_tx`, `txq.cleanup`) | +| mixed-peak | 50 RPS | 10 TPS | 60s | Consensus Health, Ledger Operations | +| cooldown | 5 RPS | — | 30s | Recovery patterns, state transitions | ### Custom Profiles @@ -120,7 +133,7 @@ Orchestrates the complete validation pipeline. Starts the telemetry stack, start # Stress test with benchmarks ./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark -# Skip Loki checks (if Phase 8 not deployed) +# Skip Loki checks (if log export is not deployed) ./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki ``` @@ -198,12 +211,12 @@ python3 tx_submitter.py --endpoint ws://localhost:6006 \ ### validate_telemetry.py -Automated validation that all expected telemetry data exists. Every metric and span is required — if it doesn't fire, the validation fails. +Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`. -- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies -- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, StatsD gauges/counters/histograms, Phase 9 OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries) to avoid false negatives from stale gauges. +- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure. +- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative. - **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki) -- **Dashboard validation**: All 15 Grafana dashboards load with panels +- **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 16** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty. ```bash # Run all validations @@ -274,9 +287,71 @@ Thresholds (configurable via environment): | Throughput impact | < 5% | BENCH_TPS_IMPACT_PCT | | Consensus impact | < 1% | BENCH_CONSENSUS_IMPACT_PCT | +Each report row is `PASS`, `FAIL`, or `INCONCLUSIVE`. The throughput and +consensus rows are ratios of the baseline, so they have nothing to report when +the baseline run measured zero — that row becomes `INCONCLUSIVE` and **counts +as a failure**, because an undefined result must never read as a pass. + +Exit codes: + +| Code | Meaning | +| ---- | --------------------------------------------------------------------------------------------------------------------------- | +| 0 | Every metric was measured and is within its threshold | +| 1 | Every metric was measured and at least one exceeded its threshold | +| 2 | The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection | + +`run-full-validation.sh` keeps the last two apart: 1 folds into its own +"checks failed" exit, 2 into its "infrastructure error" exit. A run that +measured nothing is therefore never reported as a performance regression. + +### collect_system_metrics.sh + +Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval +from the running nodes, and writes them as JSON. `benchmark.sh` calls it once +per leg; it is rarely run by hand. + +```bash +./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json +``` + +Processes are selected by matching `argv[0]`'s basename against the daemon +binary name; the pre-rename spelling is accepted too, so the sampler still +works against an older deployment. A wrapper that merely names the binary in +its arguments, and unrelated tools whose command line happens to contain the +string, are not sampled — including them diluted the CPU average and +attributed a foreign process's RSS to the node. `ps -C xrpld` is not usable +for this: xrpld renames itself, so its `comm` is `xrpld-main`. + +Selection covers the whole host, so a second xrpld from another checkout is +sampled as well. Benchmark on a machine running one cluster only. + +The output carries a `metrics_complete` flag. It is `false` when any +measurement source came back empty — no matching process, no successful RPC +probe, or a ledger sequence that never advanced — and the affected metrics are +then `0` placeholders. Since `0` clears every threshold, a `false` flag must be +read as inconclusive, never as a pass. + +Exit codes: + +| Code | Meaning | +| ---- | --------------------------------------------------------------------------------------------------------- | +| 0 | Every metric was measured; `metrics_complete` is `true` | +| 1 | Cannot run: bad arguments, no GNU `date` with `%N`, or a failed process sample. No output file is written | +| 3 | The output file was written, but `metrics_complete` is `false` | + +`benchmark.sh` treats either non-zero code — and an explicit +`"metrics_complete": false` in an otherwise successful run — as fatal, and +exits 2 rather than comparing an incomplete run. + +A nanosecond clock is required. RPC latency is graded against a 2 ms +threshold, and GNU `date +%s%N` is the only source cheap enough that the clock +does not dominate what it measures, so the script refuses to start without it. + ## Reading Validation Reports -The validation report (`validation-report.json`) is structured as: +The validation report (`validation-report.json`) is structured as follows. The +counts below are illustrative — the real total is the sum of the span, metric, +log, dashboard and parity checks for the run. ```json { @@ -304,46 +379,77 @@ Categories: - **metric**: Prometheus metric existence - **log**: Log-trace correlation checks - **dashboard**: Grafana dashboard accessibility +- **parity**: Span attributes required by the external-parity dashboard panels (validator-health, peer-quality, and friends) ## CI Integration The validation runs as a GitHub Actions workflow (`.github/workflows/telemetry-validation.yml`): -- Triggered manually or on pushes to telemetry branches +- Triggered manually (`workflow_dispatch`) or on pushes to telemetry branches. There is no cron schedule. - Builds xrpld, starts the full stack, runs load, validates -- Uploads reports as artifacts -- Posts summary to PR +- Uploads reports as artifacts (and node logs when validation did not succeed) +- Writes the validation summary and the regression-gate summary to the workflow **Step Summary** (`$GITHUB_STEP_SUMMARY`). It does **not** comment on the PR — the workflow declares no `permissions:` block and calls no GitHub API, so read the summary on the run page. + +Of the five `workflow_dispatch` inputs, only `run_benchmark` changes behaviour. +`rpc_rate`, `rpc_duration`, `tx_tps` and `tx_duration` are forwarded to +`run-full-validation.sh`, which parses them into shell variables and never reads +them again — load shape comes entirely from `--profile` and +`workload-profiles.json`. Their `description:` fields say so. ## Configuration Files -| File | Purpose | -| --------------------------------- | ------------------------------------------------------------- | -| `workload-profiles.json` | Named load profiles with phase definitions | -| `expected_spans.json` | Span inventory (names, attributes, hierarchies, config flags) | -| `expected_metrics.json` | Metric inventory — every listed metric must be present | -| `test_accounts.json` | Test account roles (keys generated at runtime) | -| `regression-metrics.json` | Metric surface for the OTel regression gate | -| `regression-thresholds.json` | Per-metric regression bounds (pct AND abs) | -| `baselines/baseline-timings.json` | Committed baseline — populated from first CI run | -| `requirements.txt` | Python dependencies | +| File | Purpose | +| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| `workload-profiles.json` | Named load profiles with phase definitions | +| `expected_spans.json` | Span inventory (names, attributes, hierarchies, config flags) | +| `expected_metrics.json` | Metric inventory — every listed metric must be present — plus the `grafana_dashboards.uids` list the dashboard check iterates | +| `test_accounts.json` | Test account roles (keys generated at runtime) | +| `regression-metrics.json` | Metric surface for the OTel regression gate | +| `regression-thresholds.json` | Per-metric regression bounds (pct AND abs) | +| `baselines/baseline-timings.json` | Committed baseline — populated from first CI run | +| `requirements.txt` | Python dependencies | ### expected_metrics.json Format ```json { + "description": "Top-level doc string — skipped by the validator.", "category_name": { "description": "Human-readable description.", "metrics": ["metric_1", "metric_2"] + }, + "grafana_dashboards": { + "uids": ["rpc-performance", "node-health"] + }, + "not_asserted": { + "description": "Why these are excluded.", + "metrics_excluded": { "metric_3": "reason" } } } ``` -Every metric listed must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it. +Every metric listed under a `metrics` array must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it. + +Three top-level keys are not metric categories: + +- `description` and `grafana_dashboards` are skipped explicitly by + `validate_metrics`. `grafana_dashboards.uids` drives the dashboard check, so + adding a dashboard to `docker/telemetry/grafana/dashboards/` does **not** put + it under the gate until its uid is added here too. +- `not_asserted` is skipped structurally: the loop reads + `category_data.get("metrics", [])`, and this group deliberately has no + `metrics` key — its entries live under `metrics_excluded` as a name-to-reason + map. It documents metrics that are emitted and dashboarded but left unasserted + because they are workload-gated or defect-gated (a check that fails on a + healthy run is worse than no check). Promote an entry into an asserted group + only after the workload is changed to guarantee it fires. ### expected_spans.json Format Each span entry defines its name, category, parent (for hierarchy validation), -required attributes, and the `config_flag` that must be enabled: +required attributes, and the `config_flag` that must be enabled. A trailing `*` +in `name` is a wildcard. The optional `"optional": true` field marks a span whose +absence is a skip rather than a failure: ```json { @@ -359,13 +465,47 @@ required attributes, and the `config_flag` that must be enabled: The orchestrator (`run-full-validation.sh`) generates node configs with: -- `[telemetry] enabled=1` with all trace categories (`trace_rpc`, `trace_consensus`, `trace_transactions`) +- `[telemetry] enabled=1` with all five trace categories: `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer`, `trace_ledger` +- `[insight] server=otel` with `endpoint=http://localhost:4318/v1/metrics` and `prefix=xrpld` — `beast::insight` metrics reach Prometheus over OTLP, because the collector declares no `statsd` receiver - `[signing_support] true` — required for `tx_submitter.py` to submit signed transactions via WebSocket -- `[ips]` (not `[ips_fixed]`) — ensures peer connections are counted in `Peer_Finder_Active_Inbound/Outbound_Peers` metrics (fixed peers are excluded from these counters by design) +- `[ips]` (not `[ips_fixed]`) — ensures peer connections are counted in the PeerFinder active-peer gauges, exported as `peer_finder_active_inbound_peers` / `peer_finder_active_outbound_peers` (fixed peers are excluded from these counters by design). The `beast::insight` group/name pair is `Peer_Finder` / `Active_Inbound_Peers`; `formatName()` lowercases it for export. -## StatsD Gauge Behaviour +## Gauge Export Behaviour -Beast::insight StatsD gauges only emit when their value _changes_ from the previous sample. This can cause two problems in the validation environment: +The harness configures each node with `[insight] server=otel` (see the +`[insight]` block generated by `run-full-validation.sh`), so `beast::insight` +gauges go through `OTelGaugeImpl` in +`src/libxrpl/beast/insight/OTelCollector.cpp`, not through the StatsD collector. +That matters for how the validator queries Prometheus. -1. **Initial-zero gauges** — if a gauge value is 0 from startup and never changes, the gauge would never emit. To address this, `StatsDGaugeImpl` initializes `m_dirty = true`, ensuring the first flush always emits the initial value. -2. **Stale gauges** — once a gauge stabilizes (e.g., peer count stays at 1), it stops emitting new data points. Prometheus marks it stale after ~5 minutes. The validation script uses the Prometheus `/api/v1/series` endpoint instead of instant queries to catch such gauges. +**How `OTelGaugeImpl` exports.** It wraps an OTel **observable** (asynchronous) +gauge. `set()` and `increment()` only store into an `std::atomic`; +nothing is exported at call time. The SDK's collection thread invokes +`gaugeCallback`, which runs the collector's hooks and then `Observe()`s whatever +the atomic currently holds. So the gauge reports **every collection cycle, +whether or not the value changed** — including a gauge that sits at 0 from +startup. There is no dirty flag on this path, and no first-flush special case is +needed. + +**Why the validator still uses `/api/v1/series`.** Two reasons survive the move +to OTLP: + +1. **Late-populating series.** A gauge or counter may not have completed the + export → collector → Prometheus-scrape pipeline by the time validation runs. + `_check_prometheus_metric` in `validate_telemetry.py` therefore polls + `/api/v1/series` (which returns anything that existed anywhere in the query + window) until the metric appears or the poll window elapses, instead of + racing a single instant query. +2. **Staleness robustness.** `/api/v1/series` does not care whether the newest + sample is inside Prometheus's ~5-minute staleness horizon, so the check + cannot be defeated by a quiet series. + +> **Note — the StatsD path is still in the tree but unused here.** If a node is +> configured with `server=statsd`, `StatsDGaugeImpl` (in +> `src/libxrpl/beast/insight/StatsDCollector.cpp`) does gate emission on a +> `dirty_` flag that is only set by `set()`/`increment()`, and it is +> initialised to `true` so the initial value is emitted on the first flush. The +> collector configs shipped in `docker/telemetry/` declare no `statsd` receiver +> (the metrics pipeline is `[otlp, spanmetrics]`) and the base +> `docker-compose.yml` keeps its StatsD UDP port commented out, so nothing in +> this harness can receive StatsD. diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md index 515a3f561f..fffea2e34d 100644 --- a/docker/telemetry/workload/baselines/README.md +++ b/docker/telemetry/workload/baselines/README.md @@ -15,7 +15,10 @@ declared in [`../regression-metrics.json`](../regression-metrics.json) and write exits 0 without gating. This is how we bootstrap the baseline. - **Populated baseline**: the comparator diffs per-metric, enforces the thresholds (regression = current exceeds baseline on BOTH the percentage AND absolute bound), - and exits non-zero on any regression. + and exits non-zero on any regression. The single exception is a baseline that is + not positive: the percentage change is undefined there, so the absolute bound + decides alone. Without that fallback the AND gate would be unreachable and a + 0 ms → 500 ms jump would be reported as "within bounds". The regression gate runs against whatever workload profile `run-full-validation.sh` was invoked with. Capture and comparison are profile-agnostic — they only read @@ -56,17 +59,70 @@ should trace back to a real CI run so variance characteristics are preserved. "profile": "", "metrics": { "span.tx.process.p99": { "value": 12.4, "unit": "ms" }, - "rpc.server_info.p95": { "value": 850.0, "unit": "us" }, "job.transaction.queued.p95": { "value": 1500.0, "unit": "us" } } } ``` +Keys follow `{category}.{name}.p{quantile}`. Only two categories are actually +produced today — `span.*` and `job.*` — because `build_query_plan()` in +`prom_queries.py` reads the `spans` and `job_queue` groups of +`regression-metrics.json`, and that file defines only those two. + Placeholder baselines additionally include `"placeholder": true`. The comparator detects this field (or an empty `metrics` object) to switch into "populate" mode instead of enforcing thresholds. Remove the `placeholder` key when pasting real captured timings. -Missing metrics (value `null`) in a captured run do not count as regressions — they -are reported separately in `regression-report.json` under `missing_in_current`. +Missing metrics (value `null`) in a captured run do not count as regressions. In +`regression-report.json`, `summary.missing_in_current` is a **count** only; the +identities are in the `metrics[]` array, as the entries whose `note` is +`"not captured in current run"`. Filter for those to see which keys went missing: + +```bash +jq -r '.metrics[] | select(.note == "not captured in current run") | .key' \ + /tmp/xrpld-validation/reports/regression-report.json +``` + This keeps the gate robust when a profile doesn't exercise every span on every run. + +## Known gap: no `rpc.*` metric can gate (FU-4) + +Per-RPC-method timings are **not** gated, and would not gate even if they were +captured. Two independent blockers: + +1. **Nothing emits an `rpc.*` key.** `build_query_plan()` in `prom_queries.py` + builds `rpc.*` entries from `cfg.get("rpc_methods", {})`, and + `regression-metrics.json` has no `rpc_methods` block — so the group resolves + to empty and no `rpc.*` key ever reaches `timings.json` or this baseline. +2. **Even a captured `rpc.*` key would silently not gate.** `resolve_thresholds()` + in `compare_to_baseline.py` maps the `rpc` category to the threshold group + `rpc_method`, but `regression-thresholds.json` defines only + `defaults.span` and `defaults.job_queue`. With no `rpc_method` block the + lookup returns `(None, None)`, which the comparator treats as "no threshold + configured" — the metric is reported but can never fail the build. + +Closing this needs **both** an `rpc_methods` group in `regression-metrics.json` +and a `defaults.rpc_method` block in `regression-thresholds.json`. Adding only +the first produces metrics that look gated in the report but are not. + +## Known exclusion: `rpc.process` is not captured + +`rpc.process` is deliberately absent from the `spans.names` list in +`regression-metrics.json`, so no `span.rpc.process.*` key appears in this +baseline. The span is created only in `ServerHandler::processRequest()` +(`src/xrpld/rpc/detail/ServerHandler.cpp:705`), which is reached only from the +HTTP/JSON-RPC session path. The harness load generator is WebSocket-only and +that path never calls `processRequest`, so the span is never emitted under any +workload profile here — `expected_spans.json` marks it `"optional": true` for +the same reason. + +While it was listed, the three quantiles were captured as `null` on every run +and the comparator short-circuited them as `"new metric (not in baseline)"` — +so a 9999 ms value would still have reported `regressed: false`. Three keys +that can never gate are worse than no keys: they inflate `summary.total` and +read as covered. + +If per-request HTTP timings are wanted, the fix is to give the harness an +HTTP/JSON-RPC load path first, then re-add `rpc.process` and bootstrap a real +baseline for it. diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json index cafd683e3c..4784953609 100644 --- a/docker/telemetry/workload/baselines/baseline-timings.json +++ b/docker/telemetry/workload/baselines/baseline-timings.json @@ -78,18 +78,6 @@ "unit": "ms", "value": 6.699999999999978 }, - "span.rpc.process.p50": { - "unit": "ms", - "value": null - }, - "span.rpc.process.p95": { - "unit": "ms", - "value": null - }, - "span.rpc.process.p99": { - "unit": "ms", - "value": null - }, "span.rpc.ws_message.p50": { "unit": "ms", "value": 0.5026522773001647 diff --git a/docker/telemetry/workload/benchmark.sh b/docker/telemetry/workload/benchmark.sh index 954322f714..6c87a6b121 100755 --- a/docker/telemetry/workload/benchmark.sh +++ b/docker/telemetry/workload/benchmark.sh @@ -17,6 +17,20 @@ # BENCH_RPC_LATENCY_IMPACT_MS=2 RPC p99 latency impact < 2ms # BENCH_TPS_IMPACT_PCT=5 Throughput impact < 5% # BENCH_CONSENSUS_IMPACT_PCT=1 Consensus round time impact < 1% +# +# Exit codes: +# 0 Every overhead metric was measured and is within its threshold. +# 1 Every overhead metric was measured and at least one exceeded its +# threshold. This is the only "telemetry is too expensive" signal. +# Also returned for a command-line usage error, which the pipeline +# cannot trigger (run-full-validation.sh passes a fixed flag list). +# 2 The overhead could not be measured at all — a missing prerequisite, a +# cluster that never reached consensus, or an incomplete metric +# collection. Nothing was compared, so nothing was breached. +# +# run-full-validation.sh depends on that split: it folds 1 into its own +# "checks failed" exit and 2 into its "infrastructure error" exit. Reporting a +# run that measured nothing as a threshold breach would be a false regression. set -euo pipefail @@ -27,11 +41,21 @@ log() { printf "\033[1;34m[BENCH]\033[0m %s\n" "$*"; } ok() { printf "\033[1;32m[BENCH]\033[0m %s\n" "$*"; } warn() { printf "\033[1;33m[BENCH]\033[0m %s\n" "$*"; } fail() { printf "\033[1;31m[BENCH]\033[0m %s\n" "$*"; } + +# Usage error. Exit 1 by shell convention; see the exit-code block above. die() { - printf "\033[1;31m[BENCH]\033[0m %s\n" "$*" >&2 + fail "$*" >&2 exit 1 } +# Fatal, and no overhead figure was produced. Exit 2 keeps exit 1 exclusively +# for a measured threshold breach, so the caller never grades a run that +# measured nothing as a performance regression. +cannot_measure() { + fail "$*" >&2 + exit 2 +} + # --------------------------------------------------------------------------- # Defaults and thresholds # --------------------------------------------------------------------------- @@ -91,11 +115,12 @@ while [ $# -gt 0 ]; do esac done -# Validate prerequisites. -[ -x "$XRPLD" ] || die "xrpld not found at $XRPLD" -command -v jq >/dev/null 2>&1 || die "jq not found" -command -v bc >/dev/null 2>&1 || die "bc not found" -command -v curl >/dev/null 2>&1 || die "curl not found" +# Validate prerequisites. A missing binary or tool means no measurement can be +# taken, which is "cannot measure", not "too slow". +[ -x "$XRPLD" ] || cannot_measure "xrpld not found at $XRPLD" +command -v jq >/dev/null 2>&1 || cannot_measure "jq not found" +command -v bc >/dev/null 2>&1 || cannot_measure "bc not found" +command -v curl >/dev/null 2>&1 || cannot_measure "curl not found" mkdir -p "$RESULTS_DIR" TIMESTAMP=$(date +%Y%m%d_%H%M%S) @@ -103,6 +128,11 @@ TIMESTAMP=$(date +%Y%m%d_%H%M%S) # --------------------------------------------------------------------------- # Node cluster management # --------------------------------------------------------------------------- + +# True while xrpld children spawned by start_cluster may still be alive. +# Read by stop_cluster so it can be called any number of times. +CLUSTER_RUNNING=false + start_cluster() { local telemetry_enabled="$1" local label="$2" @@ -115,6 +145,10 @@ start_cluster() { # Generate keys using first node. bash "$SCRIPT_DIR/generate-validator-keys.sh" "$XRPLD" "$NUM_NODES" "$WORKDIR" + # Set before the spawn loop so a failure part-way through it still gets + # cleaned up by the EXIT trap. + CLUSTER_RUNNING=true + # Build per-node configs. for i in $(seq 1 "$NUM_NODES"); do local node_dir="$WORKDIR/node$i" @@ -214,9 +248,12 @@ EOCFG echo $! >"$node_dir/xrpld.pid" done - # Wait for consensus. - log "Waiting for consensus..." - for attempt in $(seq 1 120); do + # Wait for consensus. Reaching the limit is fatal: numbers taken from a + # cluster that never got to "proposing" would silently corrupt the + # baseline-vs-telemetry comparison. + local max_wait=120 + log "Waiting for consensus (up to ${max_wait}s)..." + for attempt in $(seq 1 "$max_wait"); do local ready=0 for i in $(seq 1 "$NUM_NODES"); do local port @@ -233,8 +270,8 @@ EOCFG ok "All $NUM_NODES nodes proposing (attempt $attempt)" break fi - if [ "$attempt" -eq 120 ]; then - warn "Consensus timeout — $ready/$NUM_NODES nodes ready" + if [ "$attempt" -eq "$max_wait" ]; then + cannot_measure "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${max_wait}s" fi sleep 1 done @@ -244,6 +281,11 @@ EOCFG } stop_cluster() { + # Idempotent. The happy path calls this directly and the EXIT trap calls + # it again, so a second call must not re-kill or log a misleading message. + [ "$CLUSTER_RUNNING" = true ] || return 0 + CLUSTER_RUNNING=false + log "Stopping cluster..." for i in $(seq 1 "$NUM_NODES"); do local pidfile="$WORKDIR/node$i/xrpld.pid" @@ -251,10 +293,30 @@ stop_cluster() { kill "$(cat "$pidfile")" 2>/dev/null || true fi done - pkill -f "$WORKDIR" 2>/dev/null || true - sleep 3 + # Belt and braces for a node whose pidfile is missing or stale. Matched on + # the per-node config path — the shape start_cluster launches nodes with + # (`--conf $WORKDIR/nodeN/xrpld.cfg`) — rather than on the workdir alone. + # The loose form killed anything whose command line merely mentioned the + # workdir, including a developer's `tail -f $WORKDIR/node1/debug.log`, and + # the EXIT trap now makes this run on every exit path. + pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true + + # Guarded on purpose. This runs as the EXIT trap, where any unguarded + # failure makes `set -e` exit with that command's status and discard the + # status the script meant to report — a threshold breach would surface as + # a plain 1 and "cannot measure" would lose its 2. With every command here + # guarded, the explicit `return 0` below is reachable and authoritative. + sleep 3 || true + + return 0 } +# Reap the cluster on every exit path. Installed here rather than straight +# after argument parsing so the handler name always resolves. Without it, any +# failure between start_cluster and stop_cluster leaks the xrpld children +# along with their RPC ports (5020+) and peer ports (51250+). +trap stop_cluster EXIT + # Build RPC ports CSV string. rpc_ports_csv() { local ports="" @@ -265,6 +327,30 @@ rpc_ports_csv() { echo "$ports" } +# Collects one leg of the benchmark. +# +# The collector exits non-zero when it cannot run (1) or when a measurement +# source came back empty (3). An all-zero or partial sample set clears every +# threshold, so an incomplete leg aborts with "cannot measure" instead of being +# compared and passed. +collect_metrics() { + local label="$1" + local out_file="$2" + + local status=0 + bash "$SCRIPT_DIR/collect_system_metrics.sh" \ + "$(rpc_ports_csv)" "$DURATION" "$out_file" || status=$? + [ "$status" -eq 0 ] || + cannot_measure "$label metric collection failed (exit $status) — refusing to compare an incomplete run" + + # Only an explicit false counts. The flag is absent from older artifacts, + # and jq's "//" operator would turn a real false into the default. + local complete + complete=$(jq -r '.metrics_complete' "$out_file" 2>/dev/null || echo "null") + [ "$complete" != "false" ] || + cannot_measure "$label metrics are flagged incomplete — refusing to compare an incomplete run" +} + # --------------------------------------------------------------------------- # Run benchmark # --------------------------------------------------------------------------- @@ -276,13 +362,13 @@ log "=" # --- Baseline run --- BASELINE_FILE="$RESULTS_DIR/baseline-${TIMESTAMP}.json" start_cluster "0" "baseline" -bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$BASELINE_FILE" +collect_metrics "baseline" "$BASELINE_FILE" stop_cluster # --- Telemetry run --- TELEMETRY_FILE="$RESULTS_DIR/telemetry-${TIMESTAMP}.json" start_cluster "1" "telemetry" -bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$TELEMETRY_FILE" +collect_metrics "telemetry" "$TELEMETRY_FILE" stop_cluster # --------------------------------------------------------------------------- @@ -290,6 +376,10 @@ stop_cluster # --------------------------------------------------------------------------- log "Comparing results..." +# Written into an impact variable when its baseline could not be used. +# check_threshold turns this into an INCONCLUSIVE verdict. +INCONCLUSIVE="n/a" + read_metric() { local file="$1" local key="$2" @@ -308,20 +398,30 @@ BASE_RPC=$(read_metric "$BASELINE_FILE" "rpc_p99_ms") TELE_RPC=$(read_metric "$TELEMETRY_FILE" "rpc_p99_ms") RPC_DELTA=$(echo "scale=2; $TELE_RPC - $BASE_RPC" | bc 2>/dev/null || echo "0") +# Both impacts below are ratios of the baseline, so a non-positive baseline +# leaves them undefined. The collector writes tps=0 whenever no ledger +# advanced and read_metric defaults a missing key to 0, so this is a routine +# outcome rather than an edge case. Reporting it as "0% impact" would clear +# the threshold and hide a failed baseline run. +# +# Both expressions scale by 100 before dividing. bc truncates at "scale" after +# every operation, so dividing first would floor the ratio to 2 decimals and +# then multiply the lost precision by 100 — a real 1.25% consensus impact came +# out as exactly 1.00 and passed the 1% threshold. BASE_TPS=$(read_metric "$BASELINE_FILE" "tps") TELE_TPS=$(read_metric "$TELEMETRY_FILE" "tps") if [[ "$(echo "$BASE_TPS > 0" | bc 2>/dev/null)" = "1" ]]; then - TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) / $BASE_TPS * 100" | bc 2>/dev/null || echo "0") + TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) * 100 / $BASE_TPS" | bc 2>/dev/null || echo "0") else - TPS_IMPACT="0" + TPS_IMPACT="$INCONCLUSIVE" fi -BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_p95_ms") -TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_p95_ms") +BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_mean_ms") +TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_mean_ms") if [[ "$(echo "$BASE_CONS > 0" | bc 2>/dev/null)" = "1" ]]; then - CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) / $BASE_CONS * 100" | bc 2>/dev/null || echo "0") + CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) * 100 / $BASE_CONS" | bc 2>/dev/null || echo "0") else - CONS_IMPACT="0" + CONS_IMPACT="$INCONCLUSIVE" fi # --------------------------------------------------------------------------- @@ -329,30 +429,63 @@ fi # --------------------------------------------------------------------------- PASS_COUNT=0 FAIL_COUNT=0 +INCONCLUSIVE_COUNT=0 +# Records the verdict for one row of the report. +# +# Arguments: metric name, measured value, threshold, unit, and the name of the +# variable to write the bare verdict into. +# +# The verdict travels through that named variable and every diagnostic goes to +# stderr. Calling this through a command substitution would run it in a +# subshell, which drops the counter updates and captures the colored log line +# into the caller's variable. check_threshold() { local name="$1" local actual="$2" local threshold="$3" local unit="$4" + local result_var="$5" + + # Unusable measurement. Counted as a failure so the exit gate fires: an + # undefined result must never read as a pass. + if [ "$actual" = "$INCONCLUSIVE" ]; then + fail "$name: INCONCLUSIVE — baseline was zero or missing" >&2 + FAIL_COUNT=$((FAIL_COUNT + 1)) + INCONCLUSIVE_COUNT=$((INCONCLUSIVE_COUNT + 1)) + printf -v "$result_var" 'INCONCLUSIVE' + return + fi # Compare: actual <= threshold if [[ "$(echo "$actual <= $threshold" | bc 2>/dev/null)" = "1" ]]; then - ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS" + ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS" >&2 PASS_COUNT=$((PASS_COUNT + 1)) - echo "PASS" + printf -v "$result_var" 'PASS' else - fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL" + fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL" >&2 FAIL_COUNT=$((FAIL_COUNT + 1)) - echo "FAIL" + printf -v "$result_var" 'FAIL' fi } -CPU_RESULT=$(check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%") -MEM_RESULT=$(check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB") -RPC_RESULT=$(check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms") -TPS_RESULT=$(check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%") -CONS_RESULT=$(check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%") +# Formats a delta for the report table: appends the unit to a real number and +# leaves the INCONCLUSIVE placeholder bare. +fmt_delta() { + local value="$1" + local unit="$2" + if [ "$value" = "$INCONCLUSIVE" ]; then + printf '%s' "$value" + else + printf '%s%s' "$value" "$unit" + fi +} + +check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%" CPU_RESULT +check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB" MEM_RESULT +check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms" RPC_RESULT +check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%" TPS_RESULT +check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%" CONS_RESULT # --------------------------------------------------------------------------- # Output Markdown table @@ -373,13 +506,20 @@ cat >"$REPORT_FILE" <", - "profile": "regression", + "profile": "full-validation", "metrics": { "span.tx.process.p99": {"value": 12.4, "unit": "ms"}, - "rpc.server_info.p95": {"value": 850.0, "unit": "us"}, + "job.transaction.queued.p95": {"value": 850.0, "unit": "us"}, ... } } @@ -128,8 +128,13 @@ def main() -> int: ) parser.add_argument( "--profile", - default="regression", - help="Workload profile used during capture (metadata only)", + default="full-validation", + help=( + "Workload profile used during capture, recorded as metadata in the " + "timings file (default: full-validation). Must name a profile in " + "workload-profiles.json; run-full-validation.sh always passes this " + "explicitly." + ), ) parser.add_argument( "--min-capture-ratio", diff --git a/docker/telemetry/workload/collect_system_metrics.sh b/docker/telemetry/workload/collect_system_metrics.sh index 58c1c7b896..e7b52aba7b 100755 --- a/docker/telemetry/workload/collect_system_metrics.sh +++ b/docker/telemetry/workload/collect_system_metrics.sh @@ -17,9 +17,18 @@ # "memory_rss_mb_peak": 450.2, # "rpc_p99_ms": 15.3, # "tps": 4.8, -# "consensus_round_p95_ms": 3200, +# "consensus_round_mean_ms": 3200, +# "metrics_complete": true, # "samples": 60 # } +# +# Exit codes: +# 0 Every metric was measured; "metrics_complete" is true. +# 1 Cannot run at all: bad arguments, no GNU date with %N, or a failed +# process sample. No output file is written. +# 3 Output file was written, but at least one measurement source was empty, +# so the affected metrics are 0 placeholders and "metrics_complete" is +# false. Callers must treat this as inconclusive, never as a pass. set -euo pipefail @@ -28,6 +37,8 @@ set -euo pipefail # --------------------------------------------------------------------------- log() { printf "\033[1;34m[METRICS]\033[0m %s\n" "$*"; } ok() { printf "\033[1;32m[METRICS]\033[0m %s\n" "$*"; } +# Warnings go to stderr so they never mix into the JSON echoed on stdout. +warn() { printf "\033[1;33m[METRICS]\033[0m %s\n" "$*" >&2; } die() { printf "\033[1;31m[METRICS]\033[0m %s\n" "$*" >&2 exit 1 @@ -56,10 +67,53 @@ OUTPUT_FILE="$3" IFS=',' read -ra RPC_PORTS <<<"$RPC_PORTS_CSV" SAMPLE_INTERVAL=5 -SAMPLES=$((DURATION / SAMPLE_INTERVAL)) + +# Reject anything the sample arithmetic cannot use, instead of silently +# treating it as 0. +case "$DURATION" in + '' | *[!0-9]*) die "duration_seconds must be a positive integer, got '$DURATION'" ;; +esac + +# Normalise to base 10 once, right after the digits check. Bash arithmetic +# reads a leading zero as octal, so "08" aborted with "value too great for +# base" and "0100" was silently taken as 64. Doing it here also keeps the +# value a valid JSON number in the output below, where "08" is not. +DURATION=$((10#$DURATION)) + +# Round up, so a duration shorter than one interval still takes one sample +# rather than truncating to zero and emitting an all-zero JSON. +SAMPLES=$(((DURATION + SAMPLE_INTERVAL - 1) / SAMPLE_INTERVAL)) +if [ "$SAMPLES" -lt 1 ]; then + die "duration_seconds=$DURATION yields $SAMPLES samples; need at least 1" +fi log "Collecting metrics for ${DURATION}s (${SAMPLES} samples, ${#RPC_PORTS[@]} nodes)..." +# --------------------------------------------------------------------------- +# Nanosecond clock +# +# GNU date supports "+%s%N". BSD/macOS date has no %N and echoes it back +# literally, which used to abort the sampling loop under `set -e` and still +# exit 0 with an all-zero JSON — a silent false pass. +# +# The clock has to be cheap as well as precise, because the latency it +# measures is compared against a 2 ms threshold. Measured on a dev box: `date +# +%s%N` costs ~1.2 ms per call, forking python3 for the same value ~13 ms. +# Two calls bracket every request, so a python3 fallback would add ~26 ms of +# its own overhead to a 2 ms budget and make the number meaningless. There is +# no cheap alternative worth having, so probe once and refuse to run without +# GNU date rather than report a figure that is quietly an order of magnitude +# wrong. +# --------------------------------------------------------------------------- +if [[ ! "$(date +%s%N 2>/dev/null)" =~ ^[0-9]+$ ]]; then + die "GNU coreutils date with %N is required for RPC latency timing; this date does not support it" +fi + +# Echo the current time in nanoseconds since the epoch. +now_ns() { + date +%s%N +} + # --------------------------------------------------------------------------- # Temporary files for aggregation # --------------------------------------------------------------------------- @@ -95,40 +149,48 @@ log "Initial validated ledger seq: $INITIAL_SEQ" # Sampling loop # --------------------------------------------------------------------------- for sample in $(seq 1 "$SAMPLES"); do - # Collect CPU usage for xrpld processes. - # Uses ps to find all xrpld processes and average their CPU%. - cpu_sum=0 - cpu_count=0 - while IFS= read -r line; do - cpu_val=$(echo "$line" | awk '{print $1}') - if [ -n "$cpu_val" ] && [ "$cpu_val" != "0.0" ]; then - cpu_sum=$(echo "$cpu_sum + $cpu_val" | bc 2>/dev/null || echo "$cpu_sum") - cpu_count=$((cpu_count + 1)) - fi - done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $3}') + # Sample CPU% and RSS for the xrpld processes. One ps pass feeds both + # files, so the two numbers always come from the same instant. + # + # Selection is on argv[0]'s basename, NOT on "the command line mentions + # xrpld". The loose form matched every bystander whose command line + # happened to contain the string: this harness's own launcher (invoked as + # `--xrpld .build/xrpld`, whose argv[0] is bash), an editor's clangd, and + # any shell sitting in a directory with xrpld in its path. Measured + # against a live 5-node cluster, that pulled in 8-15 processes instead of + # 5, halved the CPU average with idle bystanders, and reported clangd's + # 1.1 GB RSS as xrpld's peak against a 5 MB threshold. `ps -C xrpld` is + # not an alternative: xrpld renames itself, so its comm is "xrpld-main" + # and -C matches nothing. "rippled" is accepted alongside "xrpld" so a + # rename of the binary cannot silently zero the collector. + # + # Scope is the whole host, as it always was: a second xrpld from another + # checkout is sampled too. Only run a benchmark on a box with one cluster. + # + # A %cpu of exactly 0.0 is a real reading and is counted — dropping idle + # samples would inflate the average — while non-numeric output is + # rejected by the pattern. An RSS of 0 is not a live process, so it + # contributes no memory sample; counting it would leave the file non-empty + # and mark a dead cluster's 0 MB peak as a complete measurement. + ps -eo %cpu=,rss=,args= | + awk -v cpu_file="$CPU_FILE" -v mem_file="$MEM_FILE" ' + $3 !~ /(^|\/)(xrpld|rippled)$/ { next } + $1 ~ /^[0-9]+(\.[0-9]+)?$/ { cpu_sum += $1; cpu_n++ } + $2 ~ /^[0-9]+$/ && $2 + 0 > 0 { printf("%.2f\n", $2 / 1024) >> mem_file } + END { if (cpu_n > 0) printf("%.2f\n", cpu_sum / cpu_n) >> cpu_file } + ' || die "process sampling failed on sample $sample/$SAMPLES" - if [ "$cpu_count" -gt 0 ]; then - cpu_avg=$(echo "scale=2; $cpu_sum / $cpu_count" | bc 2>/dev/null || echo "0") - echo "$cpu_avg" >>"$CPU_FILE" - fi - - # Collect memory RSS for xrpld processes. - while IFS= read -r line; do - rss_kb=$(echo "$line" | awk '{print $1}') - if [ -n "$rss_kb" ] && [ "$rss_kb" != "0" ]; then - rss_mb=$(echo "scale=2; $rss_kb / 1024" | bc 2>/dev/null || echo "0") - echo "$rss_mb" >>"$MEM_FILE" - fi - done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $6}') - - # Collect RPC latency from each node. + # Collect RPC latency from each node. Only a successful call is a latency + # measurement: a refused connection returns in well under a millisecond, + # and recording that as ~0 ms would pull the reported p99 down. for port in "${RPC_PORTS[@]}"; do - start_ms=$(date +%s%N) - curl -sf "http://localhost:$port" \ - -d '{"method":"server_info"}' >/dev/null 2>&1 || true - end_ms=$(date +%s%N) - latency_ms=$(((end_ms - start_ms) / 1000000)) - echo "$latency_ms" >>"$RPC_FILE" + start_ns=$(now_ns) + if curl -sf "http://localhost:$port" \ + -d '{"method":"server_info"}' >/dev/null 2>&1; then + end_ns=$(now_ns) + latency_ms=$(((end_ns - start_ns) / 1000000)) + echo "$latency_ms" >>"$RPC_FILE" + fi done # Record current validated ledger seq. @@ -153,28 +215,55 @@ done # --------------------------------------------------------------------------- log "Computing aggregated metrics..." +# Cleared by any empty measurement source. A 0 metric is otherwise +# indistinguishable from a real reading, so the flag is exported in the JSON +# and drives the exit-3 contract documented in the header. +METRICS_COMPLETE=true + # CPU average. if [ -s "$CPU_FILE" ]; then CPU_AVG=$(awk '{ sum += $1; n++ } END { if (n>0) printf "%.2f", sum/n; else print "0" }' "$CPU_FILE") else + # Now that the selector cannot match this harness's own processes, an + # empty file means no xrpld process was running for any sample. + warn "No CPU samples collected (no xrpld process matched); cpu_pct_avg is a 0 placeholder" CPU_AVG="0" + METRICS_COMPLETE=false fi # Memory peak RSS (MB). if [ -s "$MEM_FILE" ]; then MEM_PEAK=$(sort -n "$MEM_FILE" | tail -1) else + warn "No memory samples collected (no xrpld process matched); memory_rss_mb_peak is a 0 placeholder" MEM_PEAK="0" + METRICS_COMPLETE=false fi # RPC latency p99 (ms). if [ -s "$RPC_FILE" ]; then RPC_COUNT=$(wc -l <"$RPC_FILE") - P99_INDEX=$(echo "scale=0; $RPC_COUNT * 99 / 100" | bc) + # Nearest-rank p99: ceil(count * 99 / 100), clamped into [1, count]. + # Integer arithmetic avoids both the floor bias of the old bc expression + # and the bc dependency, and the lower clamp keeps sed off line address 0 + # (a file with no trailing newline makes wc -l report 0). + P99_INDEX=$(((RPC_COUNT * 99 + 99) / 100)) + if [ "$P99_INDEX" -lt 1 ]; then + P99_INDEX=1 + fi + if [ "$P99_INDEX" -gt "$RPC_COUNT" ]; then + P99_INDEX="$RPC_COUNT" + fi RPC_P99=$(sort -n "$RPC_FILE" | sed -n "${P99_INDEX}p") - [ -z "$RPC_P99" ] && RPC_P99="0" + if [ -z "$RPC_P99" ]; then + warn "RPC latency file has no line $P99_INDEX; rpc_p99_ms is a 0 placeholder" + RPC_P99="0" + METRICS_COMPLETE=false + fi else + warn "No successful RPC probes; rpc_p99_ms is a 0 placeholder" RPC_P99="0" + METRICS_COMPLETE=false fi # TPS calculation from ledger sequence advancement. @@ -193,25 +282,38 @@ LEDGER_ADVANCE=$((FINAL_SEQ - INITIAL_SEQ)) if [ "$ELAPSED" -gt 0 ] && [ "$LEDGER_ADVANCE" -gt 0 ]; then # Rough TPS: assume ~avg_txs_per_ledger * ledgers / elapsed. # Without tx count, use ledger close rate as proxy. - TPS=$(echo "scale=2; $LEDGER_ADVANCE / $ELAPSED" | bc 2>/dev/null || echo "0") + # + # awk rather than bc, because bc omits the leading zero: `scale=2` prints + # ".25", not "0.25", and a bare ".25" is not valid JSON. A value below 1 is + # the normal case here, not an edge case — ledgers close every few seconds, + # so advance/elapsed is well under 1 for any realistic window. jq happens + # to accept the malformed form, which is why it survived earlier checks, + # but a strict parser rejects the whole file. awk's %.2f always pads. + TPS=$(awk -v a="$LEDGER_ADVANCE" -v b="$ELAPSED" 'BEGIN { printf "%.2f", a / b }') else TPS="0" fi -# Consensus round time p95 (from ledger close interval). -# Approximate by looking at ledger sequence progression intervals. +# Mean inter-ledger interval in ms: DURATION / (distinct ledgers - 1) * 1000. +# +# This is a MEAN, not a percentile — the JSON key says so. It is also aliased +# by the sample loop: LEDGER_FILE gets one sequence per sample, so at a +# SAMPLE_INTERVAL of 5 s the series cannot resolve a close interval faster +# than that (a ~4 s close is invisible). Read it as a coarse trend only. if [ -s "$LEDGER_FILE" ]; then - # Calculate intervals between consecutive ledger sequences. - LEDGER_COUNT=$(wc -l <"$LEDGER_FILE") - # Rough estimate: DURATION / number_of_distinct_ledgers * 1000 ms UNIQUE_LEDGERS=$(sort -u "$LEDGER_FILE" | wc -l) + # The > 1 test also keeps the divisor below at 1 or more. if [ "$UNIQUE_LEDGERS" -gt 1 ]; then - CONSENSUS_P95=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0") + CONSENSUS_MEAN=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0") else - CONSENSUS_P95="0" + warn "Ledger seq never advanced ($UNIQUE_LEDGERS distinct); consensus_round_mean_ms is a 0 placeholder" + CONSENSUS_MEAN="0" + METRICS_COMPLETE=false fi else - CONSENSUS_P95="0" + warn "No ledger samples collected; consensus_round_mean_ms is a 0 placeholder" + CONSENSUS_MEAN="0" + METRICS_COMPLETE=false fi # --------------------------------------------------------------------------- @@ -223,7 +325,8 @@ cat >"$OUTPUT_FILE" < str: + """Classify one comparison outcome for the report and the table.""" + if regressed: + note = "REGRESSION" + elif delta < 0: + note = "improved" + else: + note = "within bounds" + if pct_change is None: + note += " (absolute bound only; baseline not positive)" + return note + + +# The regression rule, applied by compute_delta below. +# +# A regression normally requires BOTH bounds to be breached simultaneously. +# That tolerates small-value noise: a 100% increase on a 0.5 ms metric (to +# 1.0 ms) is not a regression under a 5 ms absolute bound. +# +# A non-positive baseline has no defined percentage change, so there the +# absolute bound decides alone. Requiring both bounds in that case would make +# the gate unreachable and let a 0 -> 500 ms jump pass as "within bounds". def compute_delta( key: str, baseline_entry: dict | None, @@ -183,9 +209,8 @@ def compute_delta( ) -> MetricDelta: """Compute a MetricDelta for one metric key. - A regression requires BOTH bounds to be breached simultaneously. This - tolerates small-value noise: a 100% increase on a 0.5 ms metric - (to 1.0 ms) is not a regression under a 5 ms absolute bound. + Follows the regression rule set out in the comment above, including the + non-positive-baseline exception. """ baseline = baseline_entry.get("value") if baseline_entry else None current = current_entry.get("value") if current_entry else None @@ -224,16 +249,13 @@ def compute_delta( note="no threshold configured", ) - pct_breach = pct_change is not None and pct_change > pct_threshold abs_breach = delta > abs_threshold - regressed = pct_breach and abs_breach - - if regressed: - note = "REGRESSION" - elif delta < 0: - note = "improved" + if pct_change is None: + # Baseline is not positive, so there is no percentage to compare. + # The absolute bound is the only usable signal here. + regressed = abs_breach else: - note = "within bounds" + regressed = pct_change > pct_threshold and abs_breach return MetricDelta( key=key, @@ -245,7 +267,7 @@ def compute_delta( threshold_pct=pct_threshold, threshold_abs=abs_threshold, regressed=regressed, - note=note, + note=_delta_note(regressed, delta, pct_change), ) @@ -265,7 +287,10 @@ def print_summary(deltas: list[MetricDelta]) -> None: print("=" * 72) if regressions: - print("\nRegressions (breached BOTH pct AND absolute bounds):") + print( + "\nRegressions (breached BOTH pct AND absolute bounds, or the " + "absolute bound alone where the baseline is not positive):" + ) _print_table(regressions) if improvements: diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json index 3e0242fa60..fd1193fdce 100644 --- a/docker/telemetry/workload/expected_metrics.json +++ b/docker/telemetry/workload/expected_metrics.json @@ -1,5 +1,5 @@ { - "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Sourced from the live Grafana dashboards and MetricsRegistry.cpp.", + "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY.", "spanmetrics": { "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.", "metrics": [ @@ -55,32 +55,52 @@ "total_messages_out" ] }, - "phase9_nodestore": { - "description": "Phase 9 NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.", + "nodestore_io": { + "description": "NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.", "metrics": ["nodestore_state"] }, - "phase9_cache": { - "description": "Phase 9 cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", + "cache_hit_rates": { + "description": "Cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", "metrics": ["cache_metrics"] }, - "phase9_txq": { - "description": "Phase 9 transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", + "transaction_queue": { + "description": "Transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", "metrics": ["txq_metrics"] }, - "phase9_rpc_method": { - "description": "Phase 9 per-RPC-method counters (MetricsRegistry via OTLP).", - "metrics": ["rpc_method_started_total"] + "rpc_method_detail": { + "description": "Per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.", + "metrics": [ + "rpc_method_started_total", + "rpc_method_finished_total", + "rpc_method_us_bucket", + "rpc_method_us_count", + "rpc_method_us_sum" + ] + }, + "job_queue": { + "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.", + "metrics": [ + "job_queued_total", + "job_started_total", + "job_finished_total", + "job_queued_us_bucket", + "job_queued_us_count", + "job_queued_us_sum", + "job_running_us_bucket", + "job_running_us_count", + "job_running_us_sum" + ] }, "rpc_in_flight": { "description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.", "metrics": ["rpc_in_flight_requests"] }, - "phase9_objects": { - "description": "Phase 9 counted object instances observable gauge (MetricsRegistry via OTLP).", + "object_counts": { + "description": "Counted object instances observable gauge (MetricsRegistry via OTLP).", "metrics": ["object_count"] }, - "phase9_load": { - "description": "Phase 9 fee escalation and load factor observable gauge (MetricsRegistry via OTLP).", + "load_factors": { + "description": "Fee escalation and load factor observable gauge (MetricsRegistry via OTLP).", "metrics": ["load_factor_metrics"] }, "parity_validation_agreement": { @@ -105,7 +125,7 @@ ] }, "parity_ledger_economy": { - "description": "External dashboard parity: ledger economy metrics (MetricsRegistry).", + "description": "External dashboard parity: ledger economy metrics (MetricsRegistry.cpp:1401). transaction_rate is observed on every export, in both branches of the ledger-age test (MetricsRegistry.cpp:1444-1451). base_fee_xrp is observed only inside the 'if (ledger)' guard on getValidatedLedger() (MetricsRegistry.cpp:1418-1423), and that returns validLedger_ (LedgerMaster.cpp:1569-1572), which stays null until a ledger validates — the same precondition complete_ledgers has. Both are asserted because run-full-validation.sh waits for a validated ledger before running the workload. base_fee_xrp absent while transaction_rate is present is the signature of a cluster that never validated, not of a missing metric.", "metrics": [ "ledger_economy{metric=\"base_fee_xrp\"}", "ledger_economy{metric=\"transaction_rate\"}" @@ -116,10 +136,11 @@ "metrics": ["state_tracking{metric=\"state_value\"}"] }, "parity_counters": { - "description": "External dashboard parity: monotonic counters (MetricsRegistry).", + "description": "External dashboard parity: monotonic counters (MetricsRegistry). validations_checked_total is incremented unconditionally at the top of NetworkOPsImp::recvValidation (NetworkOPs.cpp:2681), and run-full-validation.sh brings up a 5-node validator cluster, so inbound validations are guaranteed.", "metrics": [ "ledgers_closed_total", "validations_sent_total", + "validations_checked_total", "state_changes_total" ] }, @@ -197,8 +218,38 @@ "_b5_rotation_note": "WP-B5 Suspect 4 (online_delete rotation extra writes) adds one observable gauge (rotation_state, sub-series in_flight and copy_forward) and one counter (rotation_copy_node_restore_total). NONE is asserted, because the 5-node localhost harness structurally CANNOT produce any of them -- this is a documented note rather than a check that would fail CI red. Two independent reasons. First, no rotation ever runs: xrpld-validator.cfg.template sets online_delete=256 and does not set advisory_delete, so SHAMapStoreImp's gate is validatedSeq >= lastRotated + 256 (SHAMapStoreImp.cpp), which needs 256 validated ledgers; at the network's several-seconds-per-ledger close rate that is on the order of 15-20 minutes, while the full-validation profile totals 270 s of workload before Step 5 scrapes. Second, even the in_flight flag needs a rotation to have started, and copy_forward additionally needs an ARCHIVE holding data that a fetch actually reads during the rotation window -- which requires a populated, already-rotated database, exactly the condition the hypothesis says is why this slowdown never appears on a fresh node. rotation_copy_node_restore_total is narrower still: it fires only for a clean tree node reachable from the validated state map whose sole on-disk copy was removed by an EARLIER rotation, so it needs at least two rotations plus real prior data loss. Note that rotation_state publishes no series at all when online_delete is not configured, by design: MetricsRegistry::registerRotationStateGauge dynamic_casts the node store to DatabaseRotating and returns early on failure, so an absent series means 'rotation is not configured' rather than the false 'rotation is free' a zero would report. All four signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp (including the not-configured and between-rotations cases) and rendered by the ledger-sync-health panels Online-Delete Rotation Window & Copy-Forward Writes and Rotation Node Re-Store Rate. To make them assertable the harness would need a step that starts a node against a pre-populated database that has already rotated at least once, or that lowers online_delete and advisory_delete far enough to force a rotation inside the run window and then re-scrapes before teardown.", "_round_histogram_note": "consensus_round_duration_ms is a native OTel histogram recorded once per consensus round in RCLConsensus, so it needs no collector configuration -- it rides the existing OTLP -> Prometheus path. It is asserted by its Prometheus _bucket and _count series because the bare instrument name is not a series. Both are unconditional on any running cluster: every node closes ledgers continuously, so a round completes within the harness window and the histogram is populated. Absence means the record site or the explicit-bucket view regressed, not that the node was idle. The instrument carries NO labels, so exactly one series exists per node and per bucket boundary." }, + "node_health_gauges": { + "description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). server_info, build_info and db_metrics Observe unconditionally on every periodic export (build_info observes a literal 1; server_info and db_metrics read live services), so their series exist regardless of workload shape. complete_ledgers is the exception and is asserted on a narrower guarantee: its callback returns without observing when the range is empty (MetricsRegistry.cpp:1113-1114) and skips any segment that carries no '-' (:1122-1127), and a one-sequence range renders with no '-' (RangeSet.h:70-71), so it needs a complete range spanning at least two sequences. completeLedgers_ is filled by setFullLedger (LedgerMaster.cpp:862-863), which on a peered node is reached only from the publish path in doAdvance (LedgerMaster.cpp:1972) — closing a ledger is not enough, it has to validate. run-full-validation.sh waits for that before the workload starts, so on a healthy cluster the series always exists — a 5-node run yields 10 series, one start and one end per node. If this check ever fails, read the Step 3 output first: a run that logged 'No validated ledger' cannot produce this series and the cluster, not the exporter, is what broke.", + "metrics": ["server_info", "build_info", "complete_ledgers", "db_metrics"] + }, + "overlay_reduce_relay": { + "description": "Transaction reduce-relay efficiency gauge (MetricsRegistry.cpp:1354, peer-network dashboard). Backed by Overlay::txMetrics(); TxMetrics::json() emits txr_selected_cnt / txr_suppressed_cnt / txr_not_enabled_cnt unconditionally (TxMetrics.cpp:121-127), so the gauge always reports at least the selected_peers series.", + "metrics": ["reduce_relay_metrics"] + }, + "overlay_overflow": { + "description": "Job-queue transaction overflow total (MetricsRegistry.cpp:609, job-queue dashboard). An ObservableCounter that reads Overlay::getJqTransOverflow() and Observes unconditionally, so the series exists at value 0 even when no overflow occurs.", + "metrics": ["jq_trans_overflow_total"] + }, + "validation_lifetime_counters": { + "description": "Lifetime validation agreement/miss ObservableCounters (MetricsRegistry.cpp:1636, :1658, validator-health dashboard). Both callbacks reconcile the tracker and Observe unconditionally, so the series exist even on a node that has not yet agreed or missed (value 0). Only existence is asserted, never the value — validation_missed_total legitimately dominates on a non-validating node.", + "metrics": ["validation_agreements_total", "validation_missed_total"] + }, + "not_asserted": { + "description": "Emitted-and-dashboarded metrics deliberately left unasserted because they are workload-gated or defect-gated: the harness workload cannot guarantee they appear, and a check that fails on a healthy run is worse than no check. This group has no \"metrics\" key, so validate_telemetry.py skips it (validate_metrics iterates category_data.get(\"metrics\", [])). Promote an entry into an asserted group only after the workload is changed to guarantee it.", + "metrics_excluded": { + "rpc_method_errored_total": "MetricsRegistry.cpp:354, push counter — needs an RPC that returns an error. rpc_load_generator.py issues only well-formed server_info / fee / ledger / ripple_path_find calls, so no series may ever be created.", + "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.", + "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.", + "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.", + "getobject_rejected_total": "GetObjectMetricNames.h:81, emitted from PeerImp.cpp:2725/:2743 only for a TMGetObjectByHash message refused as oversize or malformed_ledgerhash. A cooperating cluster never sends one.", + "getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash — which a 5-node cluster started at genesis and already in sync may never do.", + "getobject_lookup_us": "GetObjectMetricNames.h:95, PeerImp.cpp:2929. Same lazy-creation and same inbound-request gate as getobject_request_objects.", + "getobject_lookups_total": "GetObjectMetricNames.h:100, PeerImp.cpp:2949/:2956. Same gate.", + "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate." + } + }, "grafana_dashboards": { - "description": "All Grafana dashboards that must render data (UIDs as provisioned on disk under docker/telemetry/grafana/dashboards/).", + "description": "All 15 Grafana dashboards provisioned on disk under docker/telemetry/grafana/dashboards/ (UID == file stem for every one). validate_dashboards() checks that each UID resolves via GET /api/dashboards/uid/ and reports its panel count — it verifies provisioning and loadability, not panel data. log-derived-insights is included on that basis even though its panels are Loki-backed and CI runs with --skip-loki: the dashboard itself must still provision cleanly. Its panel data is not asserted anywhere.", "uids": [ "rpc-performance", "transaction-overview", @@ -214,7 +265,8 @@ "rpc-pathfinding", "overlay-traffic-detail", "ledger-data-sync", - "ledger-sync-health" + "ledger-sync-health", + "log-derived-insights" ] } } diff --git a/docker/telemetry/workload/expected_spans.json b/docker/telemetry/workload/expected_spans.json index ec6cf28935..2ab2ea17cb 100644 --- a/docker/telemetry/workload/expected_spans.json +++ b/docker/telemetry/workload/expected_spans.json @@ -1,5 +1,5 @@ { - "description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers. Spans marked \"optional\": true are conditional \u2014 they only fire under traffic the harness may not produce (e.g. gRPC client, missing-ledger fetch, mode transitions) and are not failed when absent.", + "description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers and verified against the emitting call sites. Spans marked \"optional\": true are conditional — they only fire under traffic the harness may not produce (e.g. gRPC client, missing-ledger fetch, mode transitions) and are not failed when absent. \"parent\" is documentation only (validate_telemetry.py asserts hierarchy from parent_child_relationships, not from this field) and records the parent as the code actually produces it: null means the span is a root or an explicit freshRoot. required_attributes lists only attributes set on EVERY code path that creates the span — attributes set after an early return are described in the span's note instead, because _validate_span_attributes_otlp samples a single trace and would fail on a legitimate short-circuit path. total_unique_attributes is the size of the union of all required_attributes; total_span_types is len(spans). Span EVENTS (consensus.round phase.*/outcome.*, consensus.update_positions dispute.resolve, consensus.accept.apply tx.included) are NOT represented: validate_telemetry.py reads only span name, attributes and timestamps from Tempo, so an \"events\" key would be silently ignored. They are documented in the relevant span notes until the validator gains event support.", "spans": [ { "name": "rpc.ws_message", @@ -9,20 +9,31 @@ "config_flag": "trace_rpc", "note": "WebSocket RPC root span. The load generator uses WS, so this is the RPC entry span (not rpc.http_request, which needs an HTTP/JSON-RPC client)." }, + { + "name": "rpc.ws_upgrade", + "category": "rpc", + "parent": null, + "required_attributes": [], + "config_flag": "trace_rpc", + "optional": true, + "note": "WebSocket handshake span (ServerHandler::onHandoff, ServerHandler.cpp:272-273). A freshRoot with no attributes — only setOk() on success or recordException() on an upgrade failure. Fires once per WS connection, so the load generator produces only a handful of these at connect time; by the time validation runs after the propagation wait they may fall outside the Tempo search window. Optional for that reason, not because the code path is conditional." + }, { "name": "rpc.process", "category": "rpc", - "parent": "rpc.ws_message", + "parent": "rpc.http_request", "required_attributes": [], - "config_flag": "trace_rpc" + "config_flag": "trace_rpc", + "optional": true, + "note": "HTTP-only. Created solely in ServerHandler::processRequest() (ServerHandler.cpp:705), which is reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path that roots rpc.http_request at ServerHandler.cpp:640-641. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so this span cannot appear under the WebSocket-only harness workload." }, { "name": "rpc.command.*", "category": "rpc", - "parent": "rpc.process", + "parent": "rpc.ws_message", "required_attributes": ["command", "version", "rpc_role", "rpc_status"], "config_flag": "trace_rpc", - "note": "Wildcard \u2014 matches rpc.command.server_info, rpc.command.ledger, etc." + "note": "Wildcard — matches rpc.command.server_info, rpc.command.ledger, etc. Created as an ambient (scoped) child in rpc::doCommand / rpc::callMethod (RPCHandler.cpp:168, :271), so its parent is whichever transport span is active on the thread: rpc.ws_message on the WebSocket path (the harness workload) and rpc.process on the HTTP/JSON-RPC path." }, { "name": "rpc.http_request", @@ -113,7 +124,7 @@ "required_attributes": ["queue_size", "ledger_changed"], "config_flag": "trace_transactions", "optional": true, - "note": "Ledger-close accept loop. Fires on the consensus thread; only meaningful when the queue is non-empty." + "note": "Ledger-close accept loop (TxQ::accept, TxQ.cpp:1499). Only meaningful when the queue is non-empty. Root on BOTH call paths, verified: the consensus path (RCLConsensus.cpp:823, inside doAccept) and the switchLastClosedLedger jump path (NetworkOPs.cpp:2150). The span is a ScopedSpanGuard, so it adopts whatever OTel context is ambient — but consensus.accept and consensus.accept.apply are unscoped thread-free SpanGuards and activate() is never called outside unit tests, so no consensus span is ever the ambient parent on the JtAccept worker. ledger.build's ScopedSpanGuard has already been destroyed by the time OpenLedger::accept runs." }, { "name": "txq.accept_tx", @@ -134,7 +145,8 @@ "parent": null, "required_attributes": ["ledger_seq", "expired_count"], "config_flag": "trace_transactions", - "optional": true + "optional": true, + "note": "TxQ::processClosedLedger (TxQ.cpp:1403). Root on BOTH call paths for the same reason as txq.accept: the consensus path (RCLConsensus.cpp:950) and the switchLastClosedLedger jump path (NetworkOPs.cpp:2121) both run with no consensus span activated as ambient context." }, { "name": "consensus.round", @@ -148,7 +160,7 @@ "consensus_phase" ], "config_flag": "trace_consensus", - "note": "Root consensus span created per round. Also carries trace_strategy, previous_ledger_seq, previous_proposers, previous_round_time_ms." + "note": "Root consensus span created per round. Also carries trace_strategy, previous_ledger_seq, previous_proposers, previous_round_time_ms. Emits seven span EVENTS that this manifest cannot assert: phase.open, phase.recovery, phase.establish, phase.accepted, outcome.yes, outcome.moved_on, outcome.expired (declared ConsensusSpanNames.h:265-277; emitted RCLConsensus.cpp:1344 and via onPhaseEvent/onOutcomeEvent from Consensus.h:764, 793, 1047, 1517-1525, 1530, 1566). validate_telemetry.py reads only span name, attributes and start/end timestamps from the Tempo OTLP payload — it has no event assertion support — so adding an \"events\" key here would be silently ignored. Recorded as a note instead; asserting events needs validator support first." }, { "name": "consensus.phase.open", @@ -188,25 +200,27 @@ { "name": "consensus.update_positions", "category": "consensus", - "parent": "consensus.round", + "parent": "consensus.establish", "required_attributes": [ "converge_percent", "proposers", "disputes_count" ], - "config_flag": "trace_consensus" + "config_flag": "trace_consensus", + "note": "childSpan of establishSpanContext_ (Consensus.h:1628), so the parent is consensus.establish — not consensus.round. Also emits a dispute.resolve span EVENT per resolved dispute (Consensus.h:1697-1698), which validate_telemetry.py cannot assert (no event support)." }, { "name": "consensus.check", "category": "consensus", - "parent": "consensus.round", + "parent": "consensus.establish", "required_attributes": [ "agree_count", "disagree_count", "threshold_percent", "consensus_result" ], - "config_flag": "trace_consensus" + "config_flag": "trace_consensus", + "note": "childSpan of establishSpanContext_ (Consensus.h:1837), so the parent is consensus.establish — not consensus.round." }, { "name": "consensus.accept", @@ -228,7 +242,7 @@ "resolution_direction" ], "config_flag": "trace_consensus", - "note": "Also carries close_time_correct, close_resolution_ms, consensus_state, proposing, round_time_ms, tx_count." + "note": "Also carries close_time_correct, close_resolution_ms, consensus_state, proposing, round_time_ms, tx_count. Emits a tx.included span EVENT per transaction in the accepted set (RCLConsensus.cpp:666, with a tx_id attribute), which validate_telemetry.py cannot assert (no event support)." }, { "name": "consensus.validation.send", @@ -276,11 +290,11 @@ { "name": "consensus.mode_change", "category": "consensus", - "parent": null, + "parent": "consensus.round", "required_attributes": ["mode_old", "mode_new"], "config_flag": "trace_consensus", "optional": true, - "note": "Only fires on an operating-mode transition; a steady cluster rarely changes mode after warmup." + "note": "childSpan of roundSpanContext_ (RCLConsensus.cpp:1101), so the parent is consensus.round. Only fires on an operating-mode transition; a steady cluster rarely changes mode after warmup. A mode change outside a round leaves roundSpanContext_ invalid, which yields a null (no-op) guard rather than a root span." }, { "name": "ledger.build", @@ -401,32 +415,28 @@ "name": "peer.proposal.receive", "category": "peer", "parent": null, - "required_attributes": ["peer_id", "proposal_trusted"], - "config_flag": "trace_peer" + "required_attributes": ["peer_id"], + "config_flag": "trace_peer", + "note": "peer_id is set immediately after the freshRoot (PeerImp.cpp:1925) and is the only unconditional attribute. proposal_trusted is set at PeerImp.cpp:1953, after several early returns (stale/duplicate/self-originated proposal checks), so a single rejected proposal in the sampled trace would fail the check — it is therefore not required." }, { "name": "peer.validation.receive", "category": "peer", "parent": null, - "required_attributes": [ - "peer_id", - "validation_trusted", - "ledger_hash", - "full_validation" - ], + "required_attributes": ["peer_id", "ledger_hash", "full_validation"], "config_flag": "trace_peer", - "note": "ledger_hash and full_validation are shared with consensus.validation.send (same keys, told apart by span name)." + "note": "ledger_hash and full_validation are shared with consensus.validation.send (same keys, told apart by span name). Both are set at PeerImp.cpp:2573-2574, BEFORE the isCurrent() gate, so only a too-small or unparseable validation skips them — they stay required (and validate_telemetry.py's PARITY_SPAN_ATTRS already asserts them independently). validation_trusted is set at PeerImp.cpp:2591, after the isCurrent() early return at :2576-2584, so a single not-current validation in the sampled trace would fail the check — it is therefore not required." }, { "name": "pathfind.request", "category": "pathfind", - "parent": null, + "parent": "rpc.command.*", "required_attributes": [ "pathfind_source_account", "pathfind_dest_account" ], "config_flag": "trace_rpc", - "note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py." + "note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly." }, { "name": "pathfind.compute", @@ -471,12 +481,21 @@ "child": "rpc.process", "description": "WebSocket message contains processing span", "skip": true, - "skip_reason": "rpc.ws_message and rpc.process run on different threads (the WS handler posts a coroutine to JobQueue for processing). Span context is not propagated across the thread boundary. Requires a C++ fix to capture and forward the span context through the coroutine lambda." + "skip_reason": "This relationship does not exist in the code: rpc.process is created only in ServerHandler::processRequest() (ServerHandler.cpp:705), reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so rpc.process is never emitted at all under the WebSocket-only harness. The earlier diagnosis (cross-thread context loss needing a C++ fix) was wrong: rpc.ws_message is a deliberate freshRoot (ServerHandler.cpp:473-474) so each WS message is its own trace rather than nesting under a span leaked on a reused coroutine worker. Nothing to fix." + }, + { + "parent": "rpc.ws_message", + "child": "rpc.command.*", + "description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine)", + "skip": true, + "skip_reason": "Code-verified real, but not assertable by the current validator. _validate_parent_child() collapses the wildcard to the single literal name via child_name.replace(\"*\", \"server_info\") and samples only the 3 most recent parent traces. Each rpc.ws_message trace carries exactly one command, and server_info is 25/103 of rpc_load_generator.py's DEFAULT_WEIGHTS, so roughly 43% of healthy runs would sample three non-server_info traces and fail. Asserting this needs the validator to accept a wildcard child as a prefix match (or to raise the trace sample size); until then the relationship is documented, not enforced." }, { "parent": "rpc.process", "child": "rpc.command.*", - "description": "Processing span contains per-command span" + "description": "Processing span contains per-command span (HTTP/JSON-RPC path only)", + "skip": true, + "skip_reason": "Real relationship, but unreachable here: rpc.process only exists on the HTTP/JSON-RPC path and the harness load generator is WebSocket-only, so there are no rpc.process traces to check. The WS-path equivalent (rpc.ws_message -> rpc.command.*) is asserted above instead." }, { "parent": "ledger.build", @@ -522,6 +541,6 @@ ] }, "_conditional_attributes_note": "Five attributes documented in the 'Fresh-node sync diagnostics' table of OpenTelemetryPlan/09-data-collection-reference.md are deliberately absent from required_attributes above, because each is emitted only when its value is known and _validate_span_attributes_otlp() has no per-attribute optional flag -- listing one would fail CI red on a healthy run. ledger.acquire/peer_count is set only when finalizeAcquireSpan() is passed a peer count (InboundLedger.cpp), which the sweep and shutdown paths cannot supply. ledger_seq on the three ledger.acquire.header/.astree/.txtree phase spans is set only when seq_ != 0 (InboundLedger.cpp startPhaseSpan), and a by-hash acquire starts with seq_ == 0 and learns the sequence only when the header arrives -- so a phase that opens before the header legitimately carries no sequence. ledger_seq on ledger.serve is set only when the reply carries one (PeerImp.cpp), which an object-by-hash request does not. All five ARE indexed in the 09-reference table and rendered by the Ledger Sync Health board; the honest encoding is to document them here rather than assert a conditional attribute as required.", - "total_span_types": 47, - "total_unique_attributes": 76 + "total_span_types": 48, + "total_unique_attributes": 74 } diff --git a/docker/telemetry/workload/generate-validator-keys.sh b/docker/telemetry/workload/generate-validator-keys.sh index 7324a8bf61..48ebda8495 100755 --- a/docker/telemetry/workload/generate-validator-keys.sh +++ b/docker/telemetry/workload/generate-validator-keys.sh @@ -123,8 +123,14 @@ for i in $(seq 1 "$NUM_NODES"); do seed=$(echo "$result" | jq -r '.result.validation_seed') pubkey=$(echo "$result" | jq -r '.result.validation_public_key') + # Both fields must be present. jq -r prints the literal string "null" for + # a missing field, so an unvalidated pubkey would be written verbatim into + # validators.txt and xrpld would reject the file at startup. if [ -z "$seed" ] || [ "$seed" = "null" ]; then - die "Failed to generate key pair for node $i" + die "Failed to generate key pair for node $i: no validation_seed in response" + fi + if [ -z "$pubkey" ] || [ "$pubkey" = "null" ]; then + die "Failed to generate key pair for node $i: no validation_public_key in response" fi log " Node $i: ${pubkey:0:20}..." diff --git a/docker/telemetry/workload/prom_queries.py b/docker/telemetry/workload/prom_queries.py index b257c61241..a26a2b5459 100644 --- a/docker/telemetry/workload/prom_queries.py +++ b/docker/telemetry/workload/prom_queries.py @@ -28,6 +28,7 @@ Usage:: from __future__ import annotations +import asyncio import json import logging from dataclasses import dataclass @@ -38,6 +39,13 @@ import aiohttp logger = logging.getLogger("prom_queries") +# Instant queries run in parallel, but not all at once: a single-node +# Prometheus is easily saturated by a burst of the whole plan. With this cap +# the worst case (every query hitting the 30 s timeout) is +# ceil(len(plan) / 8) * 30 s rather than len(plan) * 30 s, which for a +# ~30-entry plan is ~2 min instead of ~15 min of a 30 min CI job. +MAX_CONCURRENT_QUERIES = 8 + @dataclass(frozen=True) class QueryEntry: @@ -147,6 +155,17 @@ async def run_query_plan( as "not yet observed" rather than as a regression. This keeps the baseline schema stable across runs with different load levels. + Queries run concurrently, at most MAX_CONCURRENT_QUERIES at a time. Keys + are emitted in plan order regardless of which query answers first. + + ``return_exceptions=True`` is what keeps the fan-out self-contained. + Without it the first escaping exception returns from ``gather`` while its + siblings keep running against a session the caller is about to close, so + the capture ends with dozens of orphaned queries and warnings from a + closed session. With it, every query is awaited before this returns, and + an unexpected failure is logged and recorded as no data -- the same + outcome _instant_query already produces for a query that fails. + Args: session: Shared aiohttp session. prom_url: Base URL of Prometheus (e.g. ``http://localhost:9090``). @@ -155,11 +174,32 @@ async def run_query_plan( Returns: Mapping from metric key to ``{"value": float|None, "unit": str}``. """ - results: dict[str, dict[str, Any]] = {} - for entry in plan: - value = await _instant_query(session, prom_url, entry.promql) - results[entry.key] = {"value": value, "unit": entry.unit} - return results + gate = asyncio.Semaphore(MAX_CONCURRENT_QUERIES) + + async def fetch(entry: QueryEntry) -> float | None: + """Run one plan entry once a query slot is free.""" + async with gate: + return await _instant_query(session, prom_url, entry.promql) + + results = await asyncio.gather( + *(fetch(entry) for entry in plan), return_exceptions=True + ) + + captured: dict[str, dict[str, Any]] = {} + for entry, result in zip(plan, results, strict=True): + value: float | None + if isinstance(result, BaseException): + logger.error( + "query for %s raised %s: %s", + entry.key, + type(result).__name__, + result, + ) + value = None + else: + value = result + captured[entry.key] = {"value": value, "unit": entry.unit} + return captured async def _instant_query( @@ -181,7 +221,10 @@ async def _instant_query( logger.warning("query HTTP %d: %s", resp.status, promql) return None body = await resp.json() - except (aiohttp.ClientError, TimeoutError) as exc: + # JSONDecodeError covers a 200 response whose body is not JSON: without it + # one malformed reply aborts the whole capture, since no caller of + # run_query_plan wraps it. + except (aiohttp.ClientError, TimeoutError, json.JSONDecodeError) as exc: logger.warning("query failed: %s — %s", promql, exc) return None diff --git a/docker/telemetry/workload/regression-metrics.json b/docker/telemetry/workload/regression-metrics.json index f475d86fd2..050ce10a4f 100644 --- a/docker/telemetry/workload/regression-metrics.json +++ b/docker/telemetry/workload/regression-metrics.json @@ -1,13 +1,13 @@ { "_description": "Metric surface for the OTel-driven regression gate. Each entry names a metric, the quantiles to capture, and how to query Prometheus. The comparator compares current run against baseline-timings.json under these exact keys.", - "_key_format": "{category}.{name}.p{quantile} (e.g. span.tx.process.p99, rpc.server_info.p95, job.transaction.queued.p95)", + "_key_format": "{category}.{name}.p{quantile} (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).", + "_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.", "spans": { "_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))", "_unit": "ms", "_quantiles": [0.5, 0.95, 0.99], "names": [ "rpc.ws_message", - "rpc.process", "tx.process", "tx.apply", "ledger.build", diff --git a/docker/telemetry/workload/requirements.txt b/docker/telemetry/workload/requirements.txt index f115de082b..062f216034 100644 --- a/docker/telemetry/workload/requirements.txt +++ b/docker/telemetry/workload/requirements.txt @@ -1,6 +1,786 @@ -# Python dependencies for Phase 10 workload tools. +# Python dependencies for the telemetry workload tools. +# +# cspell:ignore aiohappyeyeballs # # Install: pip install -r requirements.txt - -websockets>=12.0 -aiohttp>=3.9.0 +# +# Pinned with hashes so every install resolves to exactly these releases. +# Direct dependencies are websockets and aiohttp; the rest is aiohttp's +# transitive closure, which pip requires once any hash is present. +# +# Floors to keep when bumping: aiohttp >= 3.10.11 (earliest release carrying +# every 2024 aiohttp fix), websockets >= 14 (rejects a concurrent recv() with +# ConcurrencyError instead of a bare RuntimeError). +# +# Regenerate (python >= 3.11, the repo minimum, matching the universal floor): +# printf 'websockets>=17.0\naiohttp>=3.14.3\n' > requirements.in +# uv pip compile requirements.in --generate-hashes --universal \ +# --python-version 3.11 --output-file requirements.txt +aiohappyeyeballs==2.7.1 \ + --hash=sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d \ + --hash=sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472 + # via aiohttp +aiohttp==3.14.3 \ + --hash=sha256:03cd2bde3d7f085b64e549c985f4bb928cad7e8ecf5323bfca320db548d81b39 \ + --hash=sha256:041badb8f84396357c4d3ad26de6afd7a32b112f43d3c63045c0c8278cfd2043 \ + --hash=sha256:0a5ff2dfbb9ce645fa5b8ef3e02c6c0b9cc3f6030ff863d0c51fffc50cb5541b \ + --hash=sha256:0fdea2281997af69da84c77ffa6f5938a0285f21fb3887c249d67419ca865b3d \ + --hash=sha256:11fb37ef075669eee52ab1928fbf6e1741fada40409fa309ebde9607a962aebf \ + --hash=sha256:134ac5ddcf61c6fad984b9a5727d83492ada43d63471db20fb73042c13fca62f \ + --hash=sha256:152516815ef926786a0b6ae2b8f1fd2e0c71582dee0b435636865316fd4891b7 \ + --hash=sha256:1576145bdceeb92382d899751e12743a3a5b8e460a841e3e50543859e54864dc \ + --hash=sha256:16100ad3ab8d649fdfbee87602d9d2dcdca9df0b9eda8a1b5fdc0d41f96da559 \ + --hash=sha256:16ea7e24c309fb7c0bbd505d149abe4fe4dccfb8db911db7dbec0921bc889a6f \ + --hash=sha256:18c441d0a8fca6de8d1f546849b9f0ab20d435993e2c5b59562b2fae6be2f929 \ + --hash=sha256:18cb43369747b2ae007bd2655fb8e63a099c2ff1d207962943636dac989b3147 \ + --hash=sha256:1b59533861b70a2185c8f4f350f791f39d64358ef6944ce71c5240c9ec0982c9 \ + --hash=sha256:1c5281acc88b92396f88c7e1e2748f8466689df22b80170e4f51efa712fb47a8 \ + --hash=sha256:1c5ec8fb1bcc31a8466f74aaf26c345d5c386fa4bd08a3f0eb9c7a4a3fe8b5bf \ + --hash=sha256:1caa7b0d05f3e3a36f87788c59e970a7ee1cefcfcbb924a9f138c4a6551c9cb7 \ + --hash=sha256:21c016079415ed3fd676963e9793700a566d85dbbd6bfc564b9b2d209147dcc8 \ + --hash=sha256:2498f0fe69ead802f9675beca44a7c21c62fdaa4ec5145ea1c3ad6edbee29f85 \ + --hash=sha256:25bd2708db6bdf6a6630dd37bdcdfcb47c4434d22ac69c64665b802910140b30 \ + --hash=sha256:270d3dace9ca2f10f0da5d8ebe519b7a310fc6112ed916e32df5866df0888553 \ + --hash=sha256:2e1161602f45a54de2ce0905243a95f58cb42dcd378402f3697f5e0b21e9d2e7 \ + --hash=sha256:2e9878ae68e4a5f1c0abe4dd497dbc3d51946f5837b56759e2a02e78fa90ef86 \ + --hash=sha256:30402d03a7c0ff52bce290b57e564e9079fd9d0cb545c8aba73f86a103162d2e \ + --hash=sha256:33a2d7c28d33797a2e99923dffa63f83d908a19b6bf26cfe80fa790aa5e1a75a \ + --hash=sha256:362a3fd481769cac1a824514bcd86fda51c65e8fe6e051099e008fddde6db17c \ + --hash=sha256:38901a84da3ce22249f6e860bf8f90d141bcab7da090cc398f8bb58c0e44b7da \ + --hash=sha256:39aded8c7f3b935b54aab1d8d73c70ec0ee2d3ec3b943e0e86611bc150ba47f5 \ + --hash=sha256:3a26434dafe408229ff3403458ca58de24fb51936504decac49ce6755f77e59d \ + --hash=sha256:3ae5b3a59436d089b5395d910121a390feed4d00578eb95a0fd1a329fe963100 \ + --hash=sha256:3d4f72af88ac2474bb5bca640030320e3d38a0163a1d7533500e87be458eef71 \ + --hash=sha256:3f42e9b78301f11c8f861746175d8b9c1ccef713fcad9eab396e2f6db8ed4a22 \ + --hash=sha256:42a67efc36300d052fb4508a53e8b6901b9284b599ae63945c377569c5fcc1e1 \ + --hash=sha256:48d67b87db6279c044760787eb01f6413032c2e6f3ba1cafaa492b1c8e578479 \ + --hash=sha256:498c6c623134f8e09a3c4e60bcd607a0b4590dd7dbf08dd40851b27cbb520ccb \ + --hash=sha256:49f7325beb0f85ef4aef5f48f490269575f83e6e2acad00a1d80b807eb027062 \ + --hash=sha256:4e3ac92d90e92773b2362d506068e9a948192bd553e743c5b2429e28527c8661 \ + --hash=sha256:530125ee1163c4219af35dc3aa1206e541e7b31b6efc1a3f93b70a136f65d427 \ + --hash=sha256:5373dc80ad1aa2fb9ad95c83f24eef418bbda3a61375f128e5b0192e4f3f9b32 \ + --hash=sha256:53e5179d8abb5710f8e83ba207c41c8d1261fcffd4616500e15ca2b7a33be10a \ + --hash=sha256:53e7b4ce82b54a8bcc71b3b67a5cbd177ca1d7f592cbc92cd38b7349f73482db \ + --hash=sha256:543906c127fb1d929b95076db19b83fa2d46751006ff1e23b093aa5ac4d8db42 \ + --hash=sha256:54cfcdee2770dac994417cbb0ee1f3eb0e7cb6b30c79bf44f2c02ff79ec5124a \ + --hash=sha256:55bdcc472aafe2de4a253045cc128007a64f1e0264fb675791e132ea5edaa3bd \ + --hash=sha256:56f355e79f71aef2a85c80305cc915f894b170dba76de5fe84f6351939b83c06 \ + --hash=sha256:5895ef58c4620afe02fa16044f023dc4dafec08158f9d08874a46a7dbc0341b8 \ + --hash=sha256:5bcb6ff3fdab1258a192679ff1a05d44f59626430aa05cd1a9d2447423599228 \ + --hash=sha256:5f08ec777f35ee70720233b8b9811d3bb5d728137f30ac91b7457709c3261ac0 \ + --hash=sha256:614c61d478b83953e261d02bb2df750f17227cd33ef8002945bf5aebbde21919 \ + --hash=sha256:617105e2c3018ee38d0c8ce5ee3c84f621a6d8b9f723202aacaff28449ca91ee \ + --hash=sha256:6debfa7312ff9d4c124dc71d72e9a0a4b9e0879e48ba6fcb42bef5c3300289e2 \ + --hash=sha256:7041d52c3a7fa20c9e8c182b534704abb19502c8bdcbde7ab23bfda6f642394f \ + --hash=sha256:70c987b27534f9ae1a723f47ae921571d616da21d3208282bf4c52af5164ac43 \ + --hash=sha256:74ab5b6a9fb13e873e5a90946588baecaf488745e1db1a4a5c433f971f035098 \ + --hash=sha256:78253b573e6ffab5028924fc98bc281aae05445969982a10864bc360dea2016c \ + --hash=sha256:7a75aa63cbf9b21cfaf60dc2657e19df2c2867d91707d653fee171ffeedd1371 \ + --hash=sha256:8800c996b01c2772a783e3e46f3e1abd5823029adca0df54231960de9bfefa5b \ + --hash=sha256:89176250f686cb9853c0fb7ead90e639e915b84a6f43eedc2a4e7ec21f1037f0 \ + --hash=sha256:8a5fd34f7f7410d1730d5c2ba873cacb2eed3fede366feb268a70ba22581ed8f \ + --hash=sha256:8b3b60de05f3dcb6f6a00f818bb2ec781cee4de0645f59ccaf99b1d1823b6100 \ + --hash=sha256:8f2f1c4c032c7cedd7d8da6f54c97b70266c6570c3108d3fdffee7188bb70529 \ + --hash=sha256:9491196535a88924a60afd5b5f434b5b203b6cc616250878dbdb223a8f7844bc \ + --hash=sha256:9aa6e61fdf20105c4144e755bd586008ff450791d67b1c8146fdc15959c4d51c \ + --hash=sha256:9d9edccfe496b476db5f398d97b865e9a6752bcf8aec4eef8390ce20fb64bb41 \ + --hash=sha256:9fc7b5bfec6573f3ae844f457fdde5adeb713f8b8e4a81ad64fc207b49383716 \ + --hash=sha256:a0dc483c00da8b673abbb367eb6f8d8f4bcec30eb58529ea13cb42e7fd2dfa33 \ + --hash=sha256:a3a8296e7ab5c295f53f1041487cb088e1480775aafbf7fe545d93b770a0f96f \ + --hash=sha256:a3e22975f905b89a55a488c2a08f2fdb2186175349e917d48985cc468a3d4c6e \ + --hash=sha256:a4af35c443e0b1a1bd6a8af3f3485d7fda15c142751a00f3ff8090f0b93346fa \ + --hash=sha256:a94dbaae5ae27bd849c93570669bff91e0510f33a80805738e3de72a7be0447b \ + --hash=sha256:ac74facc01463f138b0da5580329cfcc82818dea5656e83ddcd11268fc12ff80 \ + --hash=sha256:ad4c8b7488d745d2ca4838ebd8ae5ba9b56341d30b1da43640e4ce87f9f49646 \ + --hash=sha256:b014a6ed7cf912e787149fdc529166d3ceabac23f26efeea3158c9aba2354e7e \ + --hash=sha256:b20032766aedf6261c7a566585a40867d092ac03a0d81592d5370ef9b054f99b \ + --hash=sha256:b2466434105a4e03113c36ec775cc2ebe6676b62eae326fa670bb607ef788c1c \ + --hash=sha256:b304db572b4368edd8dda8a2274f73156fe15558fca4a917cb8a09fc47af5963 \ + --hash=sha256:ba59d59aba08ac02fc03b0c8983ccd5ee39a199d0552ce9e6d2b4845b34d59ae \ + --hash=sha256:bd52f811e65f6fb634b1047159657c98f52b407f8efec907bcfc09da9a4c0a25 \ + --hash=sha256:bdd0e2834dce1a26c1bbe26464861e16bbe217042cbff619247c11594472518c \ + --hash=sha256:c23ec8ee9d5ab2f5421f9c7fffce208435607af27fd46d4a44e031954352838f \ + --hash=sha256:c39846c3aad97a8530c89d7a3869a8f8e9e3762c6ac0504481e5c80948f7e807 \ + --hash=sha256:c3c200cf9757edd785051dc699c7ecbec22110dbfcb3fefc7a9f9695eda8ea7a \ + --hash=sha256:c7d3a97c678d34fc5b59da671ee9cd630096ddc643e7b5a30d54a2a6f3574d3f \ + --hash=sha256:c8653fd547c93a61aadc612007790f5555cdd18946fa48cf45e26d8ea4ea473d \ + --hash=sha256:cc7cb243a68167172f48c1fd43cee91ec4b1d40cefd190edd43369d1a6bc9c82 \ + --hash=sha256:ccd4893707b3e2a13e39c90d43cf80edf2e4d0457935bcc103bf2346214c3f15 \ + --hash=sha256:cd817772b2fcf2b8c0905795318485f9ec16eae60b29feb7f4c77085311637f0 \ + --hash=sha256:cda5fd5c95ad7a125a2e8464acc78b98b94c475a3780d6aa0aa157c93f470f4d \ + --hash=sha256:cef89a58e628c4efcac3275c2d68083f82426dcdc89c1492a6f654f9f7ea6ab9 \ + --hash=sha256:d1558173930a5a8d3069cee5c92fc91c87c4dbcb099debbb3622053717145a19 \ + --hash=sha256:d6088ec9894113802bddb3c09e974929aed2c7b3a8c456219b8aab4481f1a239 \ + --hash=sha256:d6218d92e450824e9b4881f44e8c09f1853b490f9a64130801024a4793b1b3b0 \ + --hash=sha256:d77640cc618c1d99fc4f8589c0f24a730adfa54eb1e57ef7bf0c8dfb78da898c \ + --hash=sha256:d7d2deec16eeedf55f2c7cf75b521ea3856a5177e123844f8fd0f114ce252cb5 \ + --hash=sha256:db332af25642007330fca8be5c4d194caf2bea7a7fc84415aff3497af5dfee6b \ + --hash=sha256:dd54d0e8717de95939766febac482ac0474d8ac3b048115f9f2b1d23a16e7db4 \ + --hash=sha256:ddcac3c6b382e81f1dd0499199d4136b877beb4cb5ef770bbbfba56c4b8f55d2 \ + --hash=sha256:df82f3787c940c94986b34222d59c9e38843fba85139f36e85255a82ad5355a9 \ + --hash=sha256:dfa68deb2a443bdaa3ea5297b0699c1464f08aef3812b486d1348eee61b07dc0 \ + --hash=sha256:dff9461ec275f22135650d5ba4b4931a11f3958df7dfbb8db630000d4dee0883 \ + --hash=sha256:e1e74298bab6ee0d6e749ed4fd1901c7e604bdda32c03d787a2cc71c46d0433d \ + --hash=sha256:e2667f0bbe7eb6c74eae5e9691441ad186e5845ca3cff63230fc09c4e7514f5d \ + --hash=sha256:e3be98a7c30b8c25d573dafba7171d66dfb05ee6a9070fc46535464ff97700a6 \ + --hash=sha256:e568e14940c09955aa51f4e645b6daa18a581c5dcfcd73744dcc86a856e3ced3 \ + --hash=sha256:e72ee89e28d907a18f46959b4eb0bb06701cc7f8cf4366e00029e2ccfaaf5924 \ + --hash=sha256:e92eb8acc45eb6a9f4935071a77edf5b85cc6f8dfad5cd99e97653c26593cdde \ + --hash=sha256:ea05e1f97ceea523942d9b2a7d7c0359d781d683d6b043f5943a602b14da4787 \ + --hash=sha256:eac645b09bcfdf73df7536331f0678c1086ea250981118ddb5199e17ccef72bb \ + --hash=sha256:eb0495d778817619273c108784292be161a924b9f5ae5cbbc70a2caa6838250b \ + --hash=sha256:ebe8e504f058fe91223351cecd2d9d6946c9d241bb0250d898ffbdf584cc72b0 \ + --hash=sha256:ed099d105449c4f9e84f24af203cd131349d4761d8813fa7e02c32e7128cd910 \ + --hash=sha256:f0f177d1b195b9e06376cfd7d308d8a1b920909a609d03ac82a8c73bbb16d3b9 \ + --hash=sha256:f3d2669fe7dec7fc359ecdb5984b29b50d85d5d00f8c1cb61de4f4a24ee42627 \ + --hash=sha256:f4e05329faa0ea1a404b37de4f034fd2c2defcca06a68dc6745e4e56c88e8a48 \ + --hash=sha256:f53bcd52f585e1ac3e590d61434eb61f9a88c38df041b4ea126d97144344a77b \ + --hash=sha256:f55119f7bf25f49ed210f6096090715da24f2943c62102448915fde3c62877ce \ + --hash=sha256:f631fe87a6f30df5fbe6d79640b25e4cffb38c31c7fb6f10871517b84b0f8c1a \ + --hash=sha256:f8fb78a83c9e5f741ca3a68cfb455c1f5bb83b4e7249a3848b3cd78d0a8563b0 \ + --hash=sha256:fa9467a8113aa69d3d7c55a70ef0b7c636010a40993f3df9d9d0d73b3eb7ef24 \ + --hash=sha256:fd51ebf9d3a00c074df4ede271023f4d2dba289bcc740b88191872716014e3c5 + # via -r requirements.in +aiosignal==1.4.0 \ + --hash=sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e \ + --hash=sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7 + # via aiohttp +attrs==26.1.0 \ + --hash=sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309 \ + --hash=sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32 + # via aiohttp +frozenlist==1.8.0 \ + --hash=sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686 \ + --hash=sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0 \ + --hash=sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121 \ + --hash=sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd \ + --hash=sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7 \ + --hash=sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c \ + --hash=sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84 \ + --hash=sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d \ + --hash=sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b \ + --hash=sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79 \ + --hash=sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967 \ + --hash=sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f \ + --hash=sha256:13d23a45c4cebade99340c4165bd90eeb4a56c6d8a9d8aa49568cac19a6d0dc4 \ + --hash=sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7 \ + --hash=sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef \ + --hash=sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9 \ + --hash=sha256:1a7607e17ad33361677adcd1443edf6f5da0ce5e5377b798fba20fae194825f3 \ + --hash=sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd \ + --hash=sha256:1aa77cb5697069af47472e39612976ed05343ff2e84a3dcf15437b232cbfd087 \ + --hash=sha256:1b9290cf81e95e93fdf90548ce9d3c1211cf574b8e3f4b3b7cb0537cf2227068 \ + --hash=sha256:20e63c9493d33ee48536600d1a5c95eefc870cd71e7ab037763d1fbb89cc51e7 \ + --hash=sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed \ + --hash=sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b \ + --hash=sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f \ + --hash=sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25 \ + --hash=sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe \ + --hash=sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143 \ + --hash=sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e \ + --hash=sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930 \ + --hash=sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37 \ + --hash=sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128 \ + --hash=sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2 \ + --hash=sha256:332db6b2563333c5671fecacd085141b5800cb866be16d5e3eb15a2086476675 \ + --hash=sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f \ + --hash=sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746 \ + --hash=sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df \ + --hash=sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8 \ + --hash=sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c \ + --hash=sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0 \ + --hash=sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad \ + --hash=sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82 \ + --hash=sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29 \ + --hash=sha256:42145cd2748ca39f32801dad54aeea10039da6f86e303659db90db1c4b614c8c \ + --hash=sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30 \ + --hash=sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf \ + --hash=sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62 \ + --hash=sha256:48e6d3f4ec5c7273dfe83ff27c91083c6c9065af655dc2684d2c200c94308bb5 \ + --hash=sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383 \ + --hash=sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c \ + --hash=sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52 \ + --hash=sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d \ + --hash=sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1 \ + --hash=sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a \ + --hash=sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714 \ + --hash=sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65 \ + --hash=sha256:59a6a5876ca59d1b63af8cd5e7ffffb024c3dc1e9cf9301b21a2e76286505c95 \ + --hash=sha256:5a3a935c3a4e89c733303a2d5a7c257ea44af3a56c8202df486b7f5de40f37e1 \ + --hash=sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506 \ + --hash=sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888 \ + --hash=sha256:667c3777ca571e5dbeb76f331562ff98b957431df140b54c85fd4d52eea8d8f6 \ + --hash=sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41 \ + --hash=sha256:6dc4126390929823e2d2d9dc79ab4046ed74680360fc5f38b585c12c66cdf459 \ + --hash=sha256:7398c222d1d405e796970320036b1b563892b65809d9e5261487bb2c7f7b5c6a \ + --hash=sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608 \ + --hash=sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa \ + --hash=sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8 \ + --hash=sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1 \ + --hash=sha256:799345ab092bee59f01a915620b5d014698547afd011e691a208637312db9186 \ + --hash=sha256:7bf6cdf8e07c8151fba6fe85735441240ec7f619f935a5205953d58009aef8c6 \ + --hash=sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed \ + --hash=sha256:80f85f0a7cc86e7a54c46d99c9e1318ff01f4687c172ede30fd52d19d1da1c8e \ + --hash=sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52 \ + --hash=sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231 \ + --hash=sha256:8a76ea0f0b9dfa06f254ee06053d93a600865b3274358ca48a352ce4f0798450 \ + --hash=sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496 \ + --hash=sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a \ + --hash=sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3 \ + --hash=sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24 \ + --hash=sha256:940d4a017dbfed9daf46a3b086e1d2167e7012ee297fef9e1c545c4d022f5178 \ + --hash=sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695 \ + --hash=sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7 \ + --hash=sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4 \ + --hash=sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e \ + --hash=sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e \ + --hash=sha256:9ff15928d62a0b80bb875655c39bf517938c7d589554cbd2669be42d97c2cb61 \ + --hash=sha256:a6483e309ca809f1efd154b4d37dc6d9f61037d6c6a81c2dc7a15cb22c8c5dca \ + --hash=sha256:a88f062f072d1589b7b46e951698950e7da00442fc1cacbe17e19e025dc327ad \ + --hash=sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b \ + --hash=sha256:adbeebaebae3526afc3c96fad434367cafbfd1b25d72369a9e5858453b1bb71a \ + --hash=sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8 \ + --hash=sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51 \ + --hash=sha256:b37f6d31b3dcea7deb5e9696e529a6aa4a898adc33db82da12e4c60a7c4d2011 \ + --hash=sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8 \ + --hash=sha256:b4f3b365f31c6cd4af24545ca0a244a53688cad8834e32f56831c4923b50a103 \ + --hash=sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b \ + --hash=sha256:b9be22a69a014bc47e78072d0ecae716f5eb56c15238acca0f43d6eb8e4a5bda \ + --hash=sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806 \ + --hash=sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042 \ + --hash=sha256:c23c3ff005322a6e16f71bf8692fcf4d5a304aaafe1e262c98c6d4adc7be863e \ + --hash=sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b \ + --hash=sha256:c7366fe1418a6133d5aa824ee53d406550110984de7637d65a178010f759c6ef \ + --hash=sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d \ + --hash=sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567 \ + --hash=sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a \ + --hash=sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2 \ + --hash=sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0 \ + --hash=sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e \ + --hash=sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b \ + --hash=sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d \ + --hash=sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a \ + --hash=sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52 \ + --hash=sha256:d8b7138e5cd0647e4523d6685b0eac5d4be9a184ae9634492f25c6eb38c12a47 \ + --hash=sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1 \ + --hash=sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94 \ + --hash=sha256:e2de870d16a7a53901e41b64ffdf26f2fbb8917b3e6ebf398098d72c5b20bd7f \ + --hash=sha256:e4a3408834f65da56c83528fb52ce7911484f0d1eaf7b761fc66001db1646eff \ + --hash=sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822 \ + --hash=sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a \ + --hash=sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11 \ + --hash=sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581 \ + --hash=sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51 \ + --hash=sha256:ef2b7b394f208233e471abc541cc6991f907ffd47dc72584acee3147899d6565 \ + --hash=sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40 \ + --hash=sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92 \ + --hash=sha256:f57fb59d9f385710aa7060e89410aeb5058b99e62f4d16b08b91986b9a2140c2 \ + --hash=sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5 \ + --hash=sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4 \ + --hash=sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93 \ + --hash=sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027 \ + --hash=sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd + # via + # aiohttp + # aiosignal +idna==3.18 \ + --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ + --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 + # via yarl +multidict==6.7.1 \ + --hash=sha256:026d264228bcd637d4e060844e39cdc60f86c479e463d49075dedc21b18fbbe0 \ + --hash=sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9 \ + --hash=sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581 \ + --hash=sha256:067343c68cd6612d375710f895337b3a98a033c94f14b9a99eff902f205424e2 \ + --hash=sha256:08ccb2a6dc72009093ebe7f3f073e5ec5964cba9a706fa94b1a1484039b87941 \ + --hash=sha256:0b38ebffd9be37c1170d33bc0f36f4f262e0a09bc1aac1c34c7aa51a7293f0b3 \ + --hash=sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43 \ + --hash=sha256:0d17522c37d03e85c8098ec8431636309b2682cf12e58f4dbc76121fb50e4962 \ + --hash=sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1 \ + --hash=sha256:0e697826df7eb63418ee190fd06ce9f1803593bb4b9517d08c60d9b9a7f69d8f \ + --hash=sha256:10ae39c9cfe6adedcdb764f5e8411d4a92b055e35573a2eaa88d3323289ef93c \ + --hash=sha256:121a34e5bfa410cdf2c8c49716de160de3b1dbcd86b49656f5681e4543bcd1a8 \ + --hash=sha256:128441d052254f42989ef98b7b6a6ecb1e6f708aa962c7984235316db59f50fa \ + --hash=sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6 \ + --hash=sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c \ + --hash=sha256:17207077e29342fdc2c9a82e4b306f1127bf1ea91f8b71e02d4798a70bb99991 \ + --hash=sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262 \ + --hash=sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd \ + --hash=sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d \ + --hash=sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d \ + --hash=sha256:1fa6609d0364f4f6f58351b4659a1f3e0e898ba2a8c5cac04cb2c7bc556b0bc5 \ + --hash=sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3 \ + --hash=sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601 \ + --hash=sha256:24c0cf81544ca5e17cfcb6e482e7a82cd475925242b308b890c9452a074d4505 \ + --hash=sha256:25167cc263257660290fba06b9318d2026e3c910be240a146e1f66dd114af2b0 \ + --hash=sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292 \ + --hash=sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed \ + --hash=sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362 \ + --hash=sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511 \ + --hash=sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23 \ + --hash=sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2 \ + --hash=sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb \ + --hash=sha256:2e2d2ed645ea29f31c4c7ea1552fcfd7cb7ba656e1eafd4134a6620c9f5fdd9e \ + --hash=sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582 \ + --hash=sha256:38fb49540705369bab8484db0689d86c0a33a0a9f2c1b197f506b71b4b6c19b0 \ + --hash=sha256:3943debf0fbb57bdde5901695c11094a9a36723e5c03875f87718ee15ca2f4d2 \ + --hash=sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e \ + --hash=sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d \ + --hash=sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65 \ + --hash=sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a \ + --hash=sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd \ + --hash=sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d \ + --hash=sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108 \ + --hash=sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177 \ + --hash=sha256:439cbebd499f92e9aa6793016a8acaa161dfa749ae86d20960189f5398a19144 \ + --hash=sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5 \ + --hash=sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd \ + --hash=sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5 \ + --hash=sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060 \ + --hash=sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37 \ + --hash=sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56 \ + --hash=sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df \ + --hash=sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963 \ + --hash=sha256:5884a04f4ff56c6120f6ccf703bdeb8b5079d808ba604d4d53aec0d55dc33568 \ + --hash=sha256:59bc83d3f66b41dac1e7460aac1d196edc70c9ba3094965c467715a70ecb46db \ + --hash=sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118 \ + --hash=sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84 \ + --hash=sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f \ + --hash=sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889 \ + --hash=sha256:619e5a1ac57986dbfec9f0b301d865dddf763696435e2962f6d9cf2fdff2bb71 \ + --hash=sha256:65573858d27cdeaca41893185677dc82395159aa28875a8867af66532d413a8f \ + --hash=sha256:6704fa2b7453b2fb121740555fa1ee20cd98c4d011120caf4d2b8d4e7c76eec0 \ + --hash=sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7 \ + --hash=sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048 \ + --hash=sha256:6b83cabdc375ffaaa15edd97eb7c0c672ad788e2687004990074d7d6c9b140c8 \ + --hash=sha256:6d3bc717b6fe763b8be3f2bee2701d3c8eb1b2a8ae9f60910f1b2860c82b6c49 \ + --hash=sha256:6f77ce314a29263e67adadc7e7c1bc699fcb3a305059ab973d038f87caa42ed0 \ + --hash=sha256:749aa54f578f2e5f439538706a475aa844bfa8ef75854b1401e6e528e4937cf9 \ + --hash=sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59 \ + --hash=sha256:7dfb78d966b2c906ae1d28ccf6e6712a3cd04407ee5088cd276fe8cb42186190 \ + --hash=sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709 \ + --hash=sha256:7ff981b266af91d7b4b3793ca3382e53229088d193a85dfad6f5f4c27fc73e5d \ + --hash=sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c \ + --hash=sha256:844c5bca0b5444adb44a623fb0a1310c2f4cd41f402126bb269cd44c9b3f3e1e \ + --hash=sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2 \ + --hash=sha256:8affcf1c98b82bc901702eb73b6947a1bfa170823c153fe8a47b5f5f02e48e40 \ + --hash=sha256:8be1802715a8e892c784c0197c2ace276ea52702a0ede98b6310c8f255a5afb3 \ + --hash=sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee \ + --hash=sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609 \ + --hash=sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c \ + --hash=sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445 \ + --hash=sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1 \ + --hash=sha256:95922cee9a778659e91db6497596435777bd25ed116701a4c034f8e46544955a \ + --hash=sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5 \ + --hash=sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31 \ + --hash=sha256:974e72a2474600827abaeda71af0c53d9ebbc3c2eb7da37b37d7829ae31232d8 \ + --hash=sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33 \ + --hash=sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7 \ + --hash=sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca \ + --hash=sha256:98c5787b0a0d9a41d9311eae44c3b76e6753def8d8870ab501320efe75a6a5f8 \ + --hash=sha256:9b0d9b91d1aa44db9c1f1ecd0d9d2ae610b2f4f856448664e01a3b35899f3f92 \ + --hash=sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733 \ + --hash=sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429 \ + --hash=sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9 \ + --hash=sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4 \ + --hash=sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6 \ + --hash=sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2 \ + --hash=sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172 \ + --hash=sha256:a9fc4caa29e2e6ae408d1c450ac8bf19892c5fca83ee634ecd88a53332c59981 \ + --hash=sha256:aa23b001d968faef416ff70dc0f1ab045517b9b42a90edd3e9bcdb06479e31d5 \ + --hash=sha256:ac1c665bad8b5d762f5f85ebe4d94130c26965f11de70c708c75671297c776de \ + --hash=sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52 \ + --hash=sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7 \ + --hash=sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c \ + --hash=sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2 \ + --hash=sha256:b8c990b037d2fff2f4e33d3f21b9b531c5745b33a49a7d6dbe7a177266af44f6 \ + --hash=sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf \ + --hash=sha256:bb08271280173720e9fea9ede98e5231defcbad90f1624bea26f32ec8a956e2f \ + --hash=sha256:bdbf9f3b332abd0cdb306e7c2113818ab1e922dc84b8f8fd06ec89ed2a19ab8b \ + --hash=sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961 \ + --hash=sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a \ + --hash=sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3 \ + --hash=sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b \ + --hash=sha256:c524c6fb8fc342793708ab111c4dbc90ff9abd568de220432500e47e990c0358 \ + --hash=sha256:c5f0c21549ab432b57dcc82130f388d84ad8179824cc3f223d5e7cfbfd4143f6 \ + --hash=sha256:c6b3228e1d80af737b72925ce5fb4daf5a335e49cd7ab77ed7b9fdfbf58c526e \ + --hash=sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1 \ + --hash=sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c \ + --hash=sha256:c93c3db7ea657dd4637d57e74ab73de31bccefe144d3d4ce370052035bc85fb5 \ + --hash=sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53 \ + --hash=sha256:cdea2e7b2456cfb6694fb113066fd0ec7ea4d67e3a35e1f4cbeea0b448bf5872 \ + --hash=sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e \ + --hash=sha256:cf37cbe5ced48d417ba045aca1b21bafca67489452debcde94778a576666a1df \ + --hash=sha256:d4f49cb5661344764e4c7c7973e92a47a59b8fc19b6523649ec9dc4960e58a03 \ + --hash=sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8 \ + --hash=sha256:d62b7f64ffde3b99d06b707a280db04fb3855b55f5a06df387236051d0668f4a \ + --hash=sha256:d82dd730a95e6643802f4454b8fdecdf08667881a9c5670db85bc5a56693f122 \ + --hash=sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a \ + --hash=sha256:dd96c01a9dcd4889dcfcf9eb5544ca0c77603f239e3ffab0524ec17aea9a93ee \ + --hash=sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32 \ + --hash=sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3 \ + --hash=sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489 \ + --hash=sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23 \ + --hash=sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34 \ + --hash=sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75 \ + --hash=sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8 \ + --hash=sha256:eb351f72c26dc9abe338ca7294661aa22969ad8ffe7ef7d5541d19f368dc854a \ + --hash=sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d \ + --hash=sha256:f2a0a924d4c2e9afcd7ec64f9de35fcd96915149b2216e1cb2c10a56df483855 \ + --hash=sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b \ + --hash=sha256:f537b55778cd3cbee430abe3131255d3a78202e0f9ea7ffc6ada893a4bcaeea4 \ + --hash=sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4 \ + --hash=sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d \ + --hash=sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0 \ + --hash=sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba \ + --hash=sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19 + # via + # aiohttp + # yarl +propcache==0.5.2 \ + --hash=sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427 \ + --hash=sha256:04dc2390d9edbbaef7461f33322555976ffddf0b650a038649d026358714e6c5 \ + --hash=sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa \ + --hash=sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7 \ + --hash=sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a \ + --hash=sha256:0d2c9bf8528f135dbb805ce027567e09164f7efa51a2be07458a2c0420f292d0 \ + --hash=sha256:0fd59b5af35f74da48d905dcbad55449ba13be91823cb05a9bd590bbf5b61660 \ + --hash=sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94 \ + --hash=sha256:13fef48778b5a2a756523fdb781326b028ca75e32858b04f2cdd19f394564917 \ + --hash=sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42 \ + --hash=sha256:196913dea116aeb5a2ba95af4ddcb7ea85559ae07d8eee8751688310d09168c3 \ + --hash=sha256:1b31822f4474c4036bae62de9402710051d431a606d6a0f907fec79935a071aa \ + --hash=sha256:1ca071adabaab6e9219924bbe00af821f1ee7de113a9eca1cdc292de3d120f4d \ + --hash=sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33 \ + --hash=sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a \ + --hash=sha256:2293949b855ce597f2826452d17c2d545fb5622379c4ea6fdf525e9b8e8a2511 \ + --hash=sha256:26a4dca084132874e639895c3135dfad5eb20bae209f62d1aeb31b03e601c3c0 \ + --hash=sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84 \ + --hash=sha256:29cbaac5ea0212663e6845e04b5e188d5a6ae6dd919810ac835bf1d3b42c3f4c \ + --hash=sha256:29f9309a2e42b0d273be006fdb4be2d6c39a47f6f57d8fb1cf9f81481df81b66 \ + --hash=sha256:2d7aa89ebca5acc98cba9d1472d976e394782f587bad6661003602a619fd1821 \ + --hash=sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb \ + --hash=sha256:2f8ea531c794b9d6274acd4e8d2c2ebcac590a4361d27482edd3010b79f1325e \ + --hash=sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853 \ + --hash=sha256:32775082acd2d807ee3db715c7770d38767b817870acfa08c29e057f3c4d5b56 \ + --hash=sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55 \ + --hash=sha256:3b199b9b2b3d6a7edf3183ba8a9a137a22b97f7df525feb5ae1eccf026d2a9c6 \ + --hash=sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704 \ + --hash=sha256:44e488ef40dbb452700b2b1f8188934121f6648f52c295055662d2191959ff82 \ + --hash=sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f \ + --hash=sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64 \ + --hash=sha256:46088abff4cba581dea21ae0467a480526cb25aa5f3c269e909f800328bc3999 \ + --hash=sha256:4621064bbf28fa77ff64dd5d94367c04684c67d3a5bf1dff25f0cd0d98a38f3b \ + --hash=sha256:4bc8ff1feffc6a61c7002ffe84634c41b822e104990ae009f44a0834430070bb \ + --hash=sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d \ + --hash=sha256:51f96d685ab16e88cab128cd37a52c5da540809c8b879fa047731bfcb4ad35a4 \ + --hash=sha256:54adaa85a22078d1e306304a40984dc5be99d599bf3dc0a24dc98f7daeab89ab \ + --hash=sha256:552ffadf6ad409844bc5919c42a0a83d88314cedddaea0e41e80a8b8fffe881f \ + --hash=sha256:5538d2c13d93e4698af7e092b57bc7298fd35d1d58e656ae18f23ee0d0378e03 \ + --hash=sha256:5570dbcc97571c15f68068e529c92715a12f8d54030e272d264b377e22bd17a5 \ + --hash=sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba \ + --hash=sha256:583c19759d9eec1e5b69e2fbef36a7d9c326041be9746cb822d335c8cedc2979 \ + --hash=sha256:5aaa2b923c1944ac8febd6609cb373540a5563e7cbcb0fd770f75dace2eb817b \ + --hash=sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144 \ + --hash=sha256:5fcb98e7598b1ee0addab320d90f65b530297a867dbfe9de52ea838077e16e3d \ + --hash=sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e \ + --hash=sha256:66ea454f095ddf5b6b14f56c064c0941c4788be11e18d2464cf643bf7203ff67 \ + --hash=sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117 \ + --hash=sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa \ + --hash=sha256:6bf3be92233808fcd338eba0fb4d0b59ec5772af4f4ecfcec450d1bfc0f8b5eb \ + --hash=sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96 \ + --hash=sha256:6e7b8719005dd1175be4ab1cd25e9b98659a5e0347331506ec6760d2773a7fb5 \ + --hash=sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476 \ + --hash=sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191 \ + --hash=sha256:74b70780220e2dd89175ca24b81b68b67c83db499ae611e7f2313cb329801c78 \ + --hash=sha256:79aa3ff0a9b566633b642fa9caf7e21ed1c13d6feca718187873f199e1514078 \ + --hash=sha256:7afa37062e6650640e932e4cc9297d81f9f42d9944029cc386b8247dea4da837 \ + --hash=sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a \ + --hash=sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba \ + --hash=sha256:8114f28879e0904748e831c3a7774261bd9e75f49be089f389a76f959dcd13fe \ + --hash=sha256:81e3a30b0bb60caa22033dd0f8a3618d1d67356212514f62c57db75cb0ef410c \ + --hash=sha256:823581fd5cb08b12a48bfa11fe962a7916766b6170c17b028fbdf762b85eb9bf \ + --hash=sha256:85341b12b9d55bad0bded24cac341bb34289469e03a11f3f583ea1cc1db0326c \ + --hash=sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9 \ + --hash=sha256:8a90efd5777e996e42d568db9ac740b944d691e565cbfd31b2f7832f9184b2b8 \ + --hash=sha256:8b73ab70f1a3351fbc71f663b3e645af6dd0329100c353081cf69c37433fc6fe \ + --hash=sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031 \ + --hash=sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913 \ + --hash=sha256:9282fb1a3bccd038da9f768b927b24a0c753e466c086b7c4f3c6982851eefb2d \ + --hash=sha256:949c91d1a990cf3b2e8188dfcfb25005e0b834a06c63fa4ef9f360878ce21ecf \ + --hash=sha256:95f1e3f4760d404b13c9976c0229b2b49a3c8e2c62a9ce92efdd2b11ada75e3f \ + --hash=sha256:97797ebb098e670a2f92dd66f32897e30d7615b14e7f59711de23e30a9072539 \ + --hash=sha256:a0e399a2eccb91ed18721f86aa85757727400b6865c89e88934781deb9c8498b \ + --hash=sha256:a473b3440261e0c60706e732b2ed2f517857344fc21bf48fdfe211e2d98eb285 \ + --hash=sha256:a4840ab0ae0216d952f4b53dc6d0b992bfc2bedbfe360bdd9b548bc184c08959 \ + --hash=sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d \ + --hash=sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4 \ + --hash=sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f \ + --hash=sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836 \ + --hash=sha256:b05d643f944a8c3c4bd86d65ffd87bf3264b617f87791940302bc474d2ff5274 \ + --hash=sha256:b96db7141a592cbc968daf1feea83a118e6ab378af4abbc72b248c895414c22d \ + --hash=sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f \ + --hash=sha256:ba57fffe4ac99c5d30076161b5866336d97600769bad35cc68f7774b15298a4e \ + --hash=sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe \ + --hash=sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1 \ + --hash=sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a \ + --hash=sha256:c66afea89b1e43725731d2004732a046fe6fe955d51f952c3e95a7314a284a39 \ + --hash=sha256:c6844ba6364fb12f403928a82cfd295ab103a2b315c77c747b2dbe4a41894ea7 \ + --hash=sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a \ + --hash=sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164 \ + --hash=sha256:cc1177027eda740fdb152706bd215a3f124e3eea15afc39f2cb9fe351b50619e \ + --hash=sha256:cc49723e2f60d6b32a0f0b08a3fd6d13203c07f1cd9566cfce0f12a917c967a2 \ + --hash=sha256:cc6fc3cc62e8501d3ed62894425040d2728ecddb1ed072737a5c70bd537aa9f0 \ + --hash=sha256:cd416c1de191973c52ff1a12a57446bfc7642797b282d7caf2162d7d1b8aa9a0 \ + --hash=sha256:cd645f03898405cabe694fb8bc35241e3a9c332ec85627584fe3de201452b335 \ + --hash=sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568 \ + --hash=sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4 \ + --hash=sha256:d0326e2e5e1f3163fa306c834e48e8d490e5fae607a097a40c0648109b47ba80 \ + --hash=sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2 \ + --hash=sha256:d447bb0b3054be5818458fbb171208b1d9ff11eba14e18ca18b90cbb45767370 \ + --hash=sha256:d4dc37dec6c6cdad0b57881a5658fd14fbf53e333b1a86cf86559f190e1d9ec4 \ + --hash=sha256:d5a81be28596d6559f6131ef33e10200de6e17643b3c74ce03f9eb103be6ae8b \ + --hash=sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42 \ + --hash=sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a \ + --hash=sha256:decfca4c79dd53ebab484b00cc4b6717d8c369f86e74aa4ca395a64ac651495e \ + --hash=sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757 \ + --hash=sha256:e00820e192c8dbebcafb383ebbf99030895f09905e7a0eb2e0340a0bcc2bc825 \ + --hash=sha256:e4294d04a94dcab1b3bccd8b66d962dcad411a1d19414b2a41d1445f1de32ad0 \ + --hash=sha256:e59bc9e66329185b93dab73f210f1a37f81cb40f321501db8017c9aea15dba27 \ + --hash=sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf \ + --hash=sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f \ + --hash=sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d \ + --hash=sha256:f19bb891234d72535764d703bfed1153cc34f4214d5bd7150aee1eec9e8f4366 \ + --hash=sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc \ + --hash=sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c \ + --hash=sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7 \ + --hash=sha256:f814362777a9f841adddb200ecdf8f5cb1e5a3c4b7a86378edbd6ccb26edd702 \ + --hash=sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098 \ + --hash=sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751 \ + --hash=sha256:fc88b26f08d634f7bc819a7852e5214f5802641ab8d9fd5326892292eee1993e \ + --hash=sha256:fe67a3d11cd9b4efabfa45c3d00ffba2b26811442a73a581a94b67c2b5faccf6 + # via + # aiohttp + # yarl +typing-extensions==4.16.0 ; python_full_version < '3.13' \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via + # aiohttp + # aiosignal +websockets==17.0.1 \ + --hash=sha256:02ed63bf26dda9fa27df730a41f6664586c4ee05972c8fb667ce1725b3fd13d3 \ + --hash=sha256:02f0b037a737d0cb0c33866c97bcd1a0b73170dfbf42d69d8fb86f51002fd5ae \ + --hash=sha256:038cfad5d5417f8bb09295abe986029a26d22f34bda622ccc79b670efd4dab56 \ + --hash=sha256:07abc3bd196a48af476a82fd47f3f79a6a3f70937a9f930cef703cfa0c9d83b6 \ + --hash=sha256:07d78a509c3333f5908c83d7f78144ea68a6c9ec28110f5c54d81d8fcdc262c4 \ + --hash=sha256:0b52c76b8a870b141b7ca0705289452183ce7a523101954ccfe29a25986a673f \ + --hash=sha256:10b1587c599fa0f2c89154587c80e0fda98ade6c9fa8c0260a2823fb1800b685 \ + --hash=sha256:10f461191125c63902ea7394ae9e752b1b5785641850c1d365bb30b0f88bc53f \ + --hash=sha256:15920057a6b723f84734f0641403bca163a4b176e5af809ee4f0c4a1e75e9fed \ + --hash=sha256:17ac37716c0244e82c9e384c41653c090b1864c6610224ca3857e7f7b58fce10 \ + --hash=sha256:18ded646ce98cdd3c0235825b3252f1df55765ba49b616bb10282f758667b4d0 \ + --hash=sha256:1b363bfd72a52c0658a3154a4cff219f15a474b35a235057d38853bf151acce7 \ + --hash=sha256:1bdd8c4be420905dd732e00dcd669852d8128cc723efa585a0c0e51adb00a28a \ + --hash=sha256:1d4cf7e8e5b8b1fa40758ac7524843a00237b124ab217e227542cafcfeb7a946 \ + --hash=sha256:1df81d174c1561292de9e40b141cafc04f69077272f6c352afe1d743e20810df \ + --hash=sha256:20a92f78ac8250984ed459faa9ca48c285adbfc0038ddc3fdac6046990a9c9ed \ + --hash=sha256:22bd00f8bae2bccdb5dbe41e20f58ba44ca9fff0b4b561aaf39099c35da762ed \ + --hash=sha256:2437d4ca208cc0f246d3a2297ae7474b4ba18261aaf5b9c79c84c031ecf348e1 \ + --hash=sha256:246927ae9ae06ca0d42a483a4bdb80d4862e1ee5b4cab37c354a5e1ad8356448 \ + --hash=sha256:2503c7e2a5049a12d5dac917a46d5d52591283a766165b8176bb167560421b38 \ + --hash=sha256:2604de7228506b13a44a256a9d223943340c0e725af5d367dc068e192b027761 \ + --hash=sha256:28012a54510fe8301bb893ef143cec30a2780a2d3bc20b7bbdf4379d7a63945d \ + --hash=sha256:2a855b6dfe21c4d3420be265ae031829ba8ba0be0ea350d9f7c3ef30ae63ebe2 \ + --hash=sha256:2abb1ba0a5133b7d2ef3c1c9f4b0c1e8a101012dce0b594ab2b2888d9a64820e \ + --hash=sha256:2b3f3020171202b135ca078e20434977c6b2b02af647130d6980c9e39b9462e3 \ + --hash=sha256:2bc14b481e05e331811108daa1aeb41a5e237a5564ef2f02ec5a356a0f102f78 \ + --hash=sha256:2fa2cb465a131c347ba6717a78c887746e73edb1c131d01c982d6ef0d68b82e0 \ + --hash=sha256:409d93efcaa14f7a99592c5baaef5ec6ca94fba0f5aec1a86f693977c69c9c1c \ + --hash=sha256:41d6aa06b5ab832aee72fedf47a149535b121ac900b6bb4d3fe14712afac9a79 \ + --hash=sha256:49266e4488309b38783257293a38298942b9a03aa106fcb45195377a77c0c1e2 \ + --hash=sha256:4d1d99db29b5444e3982f1ce2ba8a833508ad44b2f1fbd0bd99e81d825c0b461 \ + --hash=sha256:4d41c0a1d47a478bc432b3b9068097bee1ce0c5b19327ea6f75c2ab34ab1f2fb \ + --hash=sha256:5033ffe6804dd53afafa7d08e8c3eef2d2431f34d58ca30507a8442dd04a033a \ + --hash=sha256:53b90c00bc6201ab6695c7ff51a04d0e425514c37515e9eeecd2c1b978ac6c0e \ + --hash=sha256:54cdcaa56f5d3eafd57058f0fa4a3de93a310b43a3c4699f06efc4c0bd054a5a \ + --hash=sha256:5508f38c98ac29def9e747b87543b008a58b075df6da70b2cf2e0b47073d33bb \ + --hash=sha256:55383d8177b3c99fd873ee5db0e0193f4c1dd4a3feaccf1a4a03c1b7cf539cac \ + --hash=sha256:55b12e47dcee83673a40d07686cfb6f9d6dfc285976ade9463f61d2bef3fad22 \ + --hash=sha256:5661f868ef191d33dfc6a0cc7c5b3d495f0cc8bb3f8b30d87bda8755c61c95f5 \ + --hash=sha256:57d2ee9b24b404ce75f3814f92073c0ed88106c950148d2427fe8d25ca254d1f \ + --hash=sha256:599b03beb77633bffc095334338fad79cafc2b01fbd58953838130a9ae967d7b \ + --hash=sha256:5baa9bc0dfbae8c507e51c8cf1b6d4628086f7a87bbd3a9952bd5f035451f1cc \ + --hash=sha256:5f33a649bfcb8312524173cc4bbafa7dbb236e18eee9aa31a1d324ca0ddda28c \ + --hash=sha256:6740be6d1bab69f08ab52cb15b08f76c143b6fe61c580ba62bd929f3ab7a1d42 \ + --hash=sha256:6a434e59962a4fb9016bea327e1d14d6cd67670ecfb8942b4f4a0c24036634ce \ + --hash=sha256:6db9e5bf3649ab506c6ae8a3ac85a00fb1ae3816d75962771b2df8adbc5d40d2 \ + --hash=sha256:6fd88365da261c53d3e943fb37e0d0721b9cde119f6b2e3fc84369b6ab234d63 \ + --hash=sha256:7002d5f9e1c3ddd991cdfdbfee18cc8c8b196b2445022892badacd6cb338bbbc \ + --hash=sha256:70d438268e49f1a4bd096b6b6f7010f3ab48b5db2574dbf7d8c864c46ce7a06a \ + --hash=sha256:72d7f2a5aeb4e82daa4ee18f125b4277f427033359be5c745ad709608446cc2c \ + --hash=sha256:733e3cc7171fa1b899edbe725ef9382d0e960657dc1fd933f3281ae910c01dab \ + --hash=sha256:734d20364dc2cfe03674883cafcf580b6e431c5ce42b476312b9285310230cf9 \ + --hash=sha256:759adeb5b0c5775b563254ec63b5b79089fc0045b479143a0b1b8c0ebaae1253 \ + --hash=sha256:769ce7e2acfd9a89f2bed3a9c0da229459516bbc00bd4c9e2ca492c613ae4861 \ + --hash=sha256:810cb3fb5fa6e447216f4e82d9a85cb8aed0929ae3538153ddfe8a6e3121a58d \ + --hash=sha256:81ce19c6046ace11da7001781be7317bb1dc389f399af4b2ed962190f76f9add \ + --hash=sha256:846a4a8b0833e3cad57523d9e3bd50ec8ea05ab9d06c582f82a1340ba096af5f \ + --hash=sha256:872273e629ca7e3d35f16a2dc6ede84e1d5c831e616b8277de6e4f83114e7c58 \ + --hash=sha256:8848c207049ad49d318e5f64a3d4d7bb189f8328d0d98e65647788f2a085785c \ + --hash=sha256:884af729b8ab50486acd94d9768c2b60914bf39b579ebba0a5cb73bfdfd61fd2 \ + --hash=sha256:8c07f145d0b9e90cbd96035f31fb79199aef4da1872854e36ebeb258e3d57594 \ + --hash=sha256:8cd3369e42c0246afaf9d669cfc19797e3a49e8c0a639544459c57597108b966 \ + --hash=sha256:8e387adb0c692c6b5571bdeafc8ac9d1901ea30f10309134780b16ecd35e6605 \ + --hash=sha256:90246fa9e6cb192a778ce6ce024057ec54317a894db7899c922dcdc1f4cbf6a5 \ + --hash=sha256:90973a3a00f23afdfd1c9b06fb84289bf0220f247ef8a62501a1967c7af54f7b \ + --hash=sha256:9493314a99e599163c854fb5900ad7f7ea38c5cb9d9103aa30b3c6b8181c01fa \ + --hash=sha256:9f7747d3daa41a11f25f7cca5dc988fc51da97b311bed4c9d843860f79779283 \ + --hash=sha256:a39ce3a7b0e6059be093213d637963101380157bcbad355916738fafb490698d \ + --hash=sha256:a60fa1a25cca1bcc2bf87b8d6be37a741f0a3239fb5e9cfb7a37173b68ffcf87 \ + --hash=sha256:a68e604c6d1b0338e46652e2688cbce8096ad9c03548b075fda9e2ea19a9b7dd \ + --hash=sha256:a8af570fc29cd998a921c7131c8ac81d9434466d6d25300cb12a690fb56a8a08 \ + --hash=sha256:aadc298969ad229d8e3029fc5cc751fdad286696230f9cf014e90ff9cd8e6ea0 \ + --hash=sha256:ab56439c9f74c52770690c7b2f616b3bf775cb3920453ee355ac765c032d8bbf \ + --hash=sha256:ab9f962a5b64a5c3c845d556b7dc4e6fb683f7b67179f8205e814bb2e0213ffe \ + --hash=sha256:afbce6e3f0fac32dc87c2a0d84869d1a706460d64f39f3889386413e6e4d3d26 \ + --hash=sha256:b3ff0ad440ad52dda64138f16895f66403f40192365e39b1010e889f289746b0 \ + --hash=sha256:b580794e926cab7ff42ee4371ef14e0b22cb2bb722a607f77769136468f49a3f \ + --hash=sha256:b85b960a4507b0714c0a1246d031be9118d908ee974dc085257297a955205f1d \ + --hash=sha256:b98860aefbd3d9bc8e3c7f0eefb83b11142b16110739c68cd33d3b4d6e84e536 \ + --hash=sha256:bb31f42ea095ea826463c770829aa188a86c9a5c976b1467cbbf583c811de833 \ + --hash=sha256:bc0bca48ba24c6c866847fd20478a51dd547fa0ad258dab9615c414ec534bbc0 \ + --hash=sha256:bd1470d2c53fe53269bf5619da7725d30dd9b9693f1689f7a85eab8dea734442 \ + --hash=sha256:c09e097d0e46e3c289bedab9a475ae344b70c30ff5646e46af22b4e6fdc97b21 \ + --hash=sha256:c1bec5d6a19f5fbe87e4940739cfc65e7bb53d8b353e1029b8037a1653b321bc \ + --hash=sha256:c1c118a6b0e25bfc9a6802075d748fa6321714ffbdf3c88d29d9a0e3c7386c75 \ + --hash=sha256:c23e532c8a2325a1e7486de8763a60dc43e83f01bcaeca07e3ba79652c156db1 \ + --hash=sha256:c356dbddab0a529ed7574f78f559d75a223735c321c28f6f587fbf02b11ed301 \ + --hash=sha256:c38515cb54902f7e97d0239e81ef46c4444f9475f4807fb9bbdb789b4089abcf \ + --hash=sha256:c395bda8e7d8f51a02e80261fb57127979e5c472675d9a96b2860619ad47da48 \ + --hash=sha256:c6be9cba65c65cc76dfa3d4619e359ff02a4476c74e179b215236c11a0b32345 \ + --hash=sha256:cd526c8228e759c1006c4b7c9ac71dc4e925ced1a6a6a5a8e94643709738f63e \ + --hash=sha256:cddc675ec31bca65473321f9a9794e488b43b3b8de5d02c8ef4810c5d5792163 \ + --hash=sha256:cffc84ddec6da7f447677266fee2a3c40ecc78172f00752aa1150b8a8d65df1d \ + --hash=sha256:d41e9845514754a42d1d83b2fca9d27fee2ca7b3b0bee6843ba5a9bb2b6e25ac \ + --hash=sha256:d69fd559f9f0e8a52d2fce6f04ee143f86e70df0a189cd95164eddac599e810f \ + --hash=sha256:d7d72843691f50b91127c50688df10cb72ec6f4c4b1d7e2c11ab33b16acf8e51 \ + --hash=sha256:d9aac6081513f02eac3f8caace800dbfc5c608b69e4a7bef69e414eabfc95aa1 \ + --hash=sha256:dbfae8e75b342e31fc6fd1a8bbb393b7cbb91d6cfd581650300a94381e7b7e2b \ + --hash=sha256:e8208f2729cba030ff872a92064c97584eeb9502f53d32a05a0f05d5a17ca6c6 \ + --hash=sha256:e95e321d0d763f2b6633512605f6112ebd70d5746f3ce05c941909d4a25233f2 \ + --hash=sha256:e98ec9ec61cce5bc4b8b218322ad090b0994eb060bb04da704c62ef0a3d864e6 \ + --hash=sha256:eab6de8a98b9a7772cf686d00b4de439fc7efb8ab05ae106ef227291d06f87c5 \ + --hash=sha256:efe0ae052a8d023b87198921e8a7ce1dc7768816bcd2fbc20df171ac73a04891 \ + --hash=sha256:f11a398d8170b7ac5000baf7f258dcda579ef3ea744e0cc6a165e0dfbc0d3198 \ + --hash=sha256:f3fd9a1f87f8f0f3f8e9f9bd0195f7516562d13f5b178db8c5784d1f60b60bed \ + --hash=sha256:f47b0815af3948ec6a440b3afa02f05b18cc0939549e91b5c677b5d9c2c8472a \ + --hash=sha256:f991247276797d0c61ab7770bc9791eadc16f683b4d83517f624932adc1a8bab \ + --hash=sha256:ffad64ce7ad3703d652a3fd9af26238377d24ce52c6ad8ff35d26d82f61f493f + # via -r requirements.in +yarl==1.24.5 \ + --hash=sha256:0055afc45e864b92729ac7600e2d102c17bef060647e74bca75fa84d66b9ff36 \ + --hash=sha256:0465ec8cedc2349b97a6b595ace64084a50c6e839eca40aa0626f38b8350e331 \ + --hash=sha256:0ebfaffe1a16cb72141c8e09f18cc76856dbe58639f393a4f2b26e474b96b871 \ + --hash=sha256:16a2f5010280020e90f5330257e6944bc33e73593b136cc5a241e6c1dc292498 \ + --hash=sha256:17f57620f5475b3c69109376cc87e42a7af5db13c9398e4292772a706ff10780 \ + --hash=sha256:2120b96872df4a117cde97d270bac96aea7cc52205d305cf4611df694a487027 \ + --hash=sha256:240cbec09667c1fed4c6cd0060b9ec57332427d7441289a2ed8875dc9fb2b224 \ + --hash=sha256:24e861e9630e0daddcb9191fb187f60f034e17a4426f8101279f0c475cd74144 \ + --hash=sha256:2729fcfc4f6a596fb0c50f32090400aa9367774ac296a00387e65098c0befa76 \ + --hash=sha256:2c1fe720934a16ea8e7146175cba2126f87f54912c8c5435e7f7c7a51ef808d3 \ + --hash=sha256:2cabe6546e41dabe439999a23fcb5246e0c3b595b4315b96ef755252be90caeb \ + --hash=sha256:2dbe06fc16bc91502bca713704022182e5729861ae00277c3a23354b40929740 \ + --hash=sha256:3363fcc96e665878946ad7a106b9a13eac0541766a690ef287c0232ac768b6ec \ + --hash=sha256:377fe3732edbaf78ee74efdf2c9f49f6e99f20e7f9d2649fda3eb4badd77d76e \ + --hash=sha256:3ac6aff147deb9c09461b2d4bbdf6256831198f5d8a23f5d37138213090b6d8a \ + --hash=sha256:3f45789ce415a7ec0820dc4f82925f9b5f7732070be1dec1f5f23ec381435a24 \ + --hash=sha256:4103b77b8a8225e413107d2349b65eb3c1c52627b5cc5c3c4c1c6a798b218950 \ + --hash=sha256:4377407001ca3c057773f44d8ddd6358fa5f691407c1ba92210bd3cf8d9e4c95 \ + --hash=sha256:46c2f213e23a04b93a392942d782eb9e413e6ef6bf7c8c53884e599a5c174dcb \ + --hash=sha256:47e98aab9d8d82ff682e7b0b5dded33bf138a32b817fcf7fa3b27b2d7c412928 \ + --hash=sha256:4a36f9becdd4c5c52a20c3e9484128b070b1dcfc8944c006f3a528295a359a9c \ + --hash=sha256:4af7b7e1be0a69bee8210735fe6dcfc38879adfac6d62e789d53ba432d1ffa41 \ + --hash=sha256:4d97a951a81039050e45f04e96689b58b8243fa5e62aa14fe67cb6075300885e \ + --hash=sha256:4db9aecb141cb7a5447171b57aa1ed3a8fee06af40b992ffc31206c0b0121550 \ + --hash=sha256:53e549287ef628fecba270045c9701b0c564563a9b0577d24a4ec75b8ab8040f \ + --hash=sha256:56b149b22de33b23b0c6077ab9518c6dcb538ad462e1830e68d06591ccf6e38b \ + --hash=sha256:570fec8fbd22b032733625f03f10b7ff023bc399213db15e72a7acaef28c2f4e \ + --hash=sha256:5b8ee53be440a0cffc991a27be3057e0530122548dbe7c0892df08822fce5ede \ + --hash=sha256:5ba4f78df2bcc19f764a4b26a8a4f5049c110090ad5825993aacb052bf8003ad \ + --hash=sha256:5c55256dee8f4b27bfbf636c8363383c7c8db7890c7cba5217d7bd5f5f21dab6 \ + --hash=sha256:5c88e5815a49d289e599f3513aa7fde0bc2092ff188f99c940f007f90f53d104 \ + --hash=sha256:5fede79c6f73ff2c3ef822864cb1ada23196e62756df53bc6231d351a49516a2 \ + --hash=sha256:65be18ec59496c13908f02a2472751d9ef840b4f3fb5726f129306bf6a2a7bba \ + --hash=sha256:66410eb6345d467151934b49bfa70fb32f5b35a6140baa40ad97d6436abea2e9 \ + --hash=sha256:665b0a2c463cc9423dd647e0bfd9f4ccc9b50f768c55304d5e9f80b177c1de12 \ + --hash=sha256:6b8536851f9f65e7f00c7a1d49ba7f2be0ffe2c11555367fc9f50d9f842410a1 \ + --hash=sha256:6c95b17fe34ed802f17e205112e6e10db92275c34fee290aa9bdc55a9c724027 \ + --hash=sha256:6e73e7fe93f17a7b191f52ec9da9dd8c06a8fe735a1ecbd13b97d1c723bff385 \ + --hash=sha256:6efbccc3d7f75d5b03105172a8dc86d82ba4da86817952529dd93185f4a88be2 \ + --hash=sha256:709f1efed56c4a145793c046cd4939f9959bcd818979a787b77d8e09c57a0840 \ + --hash=sha256:79af890482fc94648e8cde4c68620378f7fef60932710fa17a66abc039244da2 \ + --hash=sha256:7bcbe0fcf850eae67b6b01749815a4f7161c560a844c769ad7b48fcd99f791c4 \ + --hash=sha256:7c0494a31a1ac5461a226e7947a9c9b78c44e1dc7185164fa7e9651557a5d9bc \ + --hash=sha256:7ce27823052e2013b597e0c738b13e7e36b8ccb9400df8959417b052ab0fd92c \ + --hash=sha256:7f72c74aa99359e27a2ee8d6613fefa28b5f76a983c083074dfc2aaa4ab46213 \ + --hash=sha256:7fa5e51397466ea7e98de493fa2ff1b8193cfef8a7b0f9b4842f92d342df0dba \ + --hash=sha256:82632daed195dcc8ea664e8556dc9bdbd671960fb3776bd92806ce05792c2448 \ + --hash=sha256:82f75e05912e84b7a0fe57075d9c59de3cb352b928330f2eb69b2e1f54c3e1f0 \ + --hash=sha256:841f0852f48fefea3b12c9dfec00704dfa3aef5215d0e3ce564bb3d7cd8d57c6 \ + --hash=sha256:874019bd513008b009f58657134e5d0c5e030b3559bd0553976837adf52fe966 \ + --hash=sha256:88f50c94e21a0a7f14042c015b0eba1881af78562e7bf007e0033e624da59750 \ + --hash=sha256:89a1bbb58e0e3f7a283653d854b1e95d65e5cfd4af224dac5f02629ec1a3e621 \ + --hash=sha256:8a6987eaad834cb32dd57d9d582225f0054a5d1af706ccfbbdba735af4927e13 \ + --hash=sha256:8ac73abdc7ab75610f95a8fd994c6457e87752b02a63987e188f937a1fc180f0 \ + --hash=sha256:8ccf9aca873b767977c73df497a85dbedee4ee086ae9ae49dc461333b9b79f58 \ + --hash=sha256:90333fd89b43c0d08ac85f3f1447593fc2c66de18c3d6378d7125ea118dc7a54 \ + --hash=sha256:92ab3e11448f2ff7bf53c5a26eff0edc086898ec8b21fb154b85839ce1d88075 \ + --hash=sha256:9335a099ad87287c37fe5d1a982ff392fa5efe5d14b40a730b1ec1d6a41382b4 \ + --hash=sha256:96d30286dd02679e32a39aa8f0b7498fc847fcda46cfc09df5513e82ce252440 \ + --hash=sha256:9baafc71b04f8f4bb0703b21d6fc9f0c30b346c636a532ff16ec8491a5ea4b1f \ + --hash=sha256:9d1216a7f6f77836617dba35687c5b78a4170afc3c3f18fc788f785ba26565c4 \ + --hash=sha256:9d399bdcfb4a0f659b9b3788bbc89babe63d9a6a65aacdf4d4e7065ff2e6316c \ + --hash=sha256:9e4e16c73d717c5cf27626c524d0a2e261ad20e46932b2670f64ad5dde23e26f \ + --hash=sha256:9f4d8cf085a4c6a40fb97ea0f46938a8df43c85d31f9d45e2a8867ea9293790d \ + --hash=sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7 \ + --hash=sha256:a3732e66413163e72508da9eff9ce9d2846fde51fae45d3605393d3e6cd303e9 \ + --hash=sha256:a4582acf7ef76482f6f511ebaf1946dae7f2e85ec4728b81a678c01df63bd723 \ + --hash=sha256:a61834fb15d81322d872eaafd333838ae7c9cea84067f232656f75965933d047 \ + --hash=sha256:a7cff474ab7cd149765bb784cf6d78b32e18e20473fb7bda860bce98ab58e9da \ + --hash=sha256:a8fe66b8f300da93798025a785a5b90b42f3810dc2b72283ff84a41aaaebc293 \ + --hash=sha256:a929d878fec099030c292803b31e5d5540a7b6a31e6a3cc76cb4685fc2a2f51b \ + --hash=sha256:ad5d8201d310b031e6cd839d9bac2d4e5a01533ce5d3d5b50b7de1ef3af1de61 \ + --hash=sha256:af3aefa655adb5869491fa907e652290386800ae99cc50095cba71e2c6aefdca \ + --hash=sha256:c0ebc836c47a6477e182169c6a476fc691d12b518894bf7dd2572f0d59f1c7ed \ + --hash=sha256:c687ed078e145f5fd53a14854beff320e1d2ab76df03e2009c98f39a0f68f39a \ + --hash=sha256:cbb833ccacdb5519eff9b8b71ee618cc2801c878e77e288775d77c3a2ced858a \ + --hash=sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688 \ + --hash=sha256:d46b86567dd4e248c6c159fcbcdcce01e0a5c8a7cd2334a0fff759d0fa075b16 \ + --hash=sha256:d693396e5aea78db03decd60aec9ece16c9b40ba00a587f089615ff4e718a81d \ + --hash=sha256:d897129df1a22b12aeed2c2c98df0785a2e8e6e0bde87b389491d0025c187077 \ + --hash=sha256:daba5e594f06114e37db186efd2dd916609071e59daca901a0a2e71f02b142ce \ + --hash=sha256:dd625535328fd9882374356269227670189adfcc6a2d90284f323c05862eecbd \ + --hash=sha256:e006d3a974c4ee19512e5f058abedb6eef36a5e553c14812bdeba1758d812e6d \ + --hash=sha256:e1ae548a9d901adca07899a4147a7c826bbcc06239d3ce9a59f57886a28a4c88 \ + --hash=sha256:e2935f8c39e3b03e83519292d78f075189978f3f4adc15a78144c7c8e2a1cba5 \ + --hash=sha256:e42d75862735da90e7fc5a7b23db0c976f737113a54b3c9777a9b665e9cbff75 \ + --hash=sha256:e7d42c531243450ef0d4d9c172e7ed6ef052640f195629065041b5add4e058d1 \ + --hash=sha256:e81b83143bee16329c23db3c1b2d82b29892fcbcb849186d2f6e98a5abe9a57f \ + --hash=sha256:e8ffa78582120024f476a611d7befc123cee59e47e8309d470cf667d806e613b \ + --hash=sha256:ebb0ec7f17803063d5aeb982f3b1bd2b2f4e4fae6751226cbd6ba1fcfe9e63ff \ + --hash=sha256:f08c7513ecef5aad65687bfdf6bc601ae9fccd04a42904501f8f7141abad9eb9 \ + --hash=sha256:f0a658a6d3fafee5c6f63c58f3e785c8c43c93fbc02bf9f2b6663f8185e0971f \ + --hash=sha256:f0e466ed7511fe9d459a819edbc6c2585c0b6eabde9fa8a8947552468a7a6ef0 \ + --hash=sha256:f141474e85b7e54998ec5180530a7cda99ab29e282fa50e0756d89981a9b43c5 \ + --hash=sha256:f4239bbec5a3577ddb49e4b50aeb32d8e5792098262ae2f63723f916a29b1a25 \ + --hash=sha256:f540c013589084679a6c7fac07096b10159737918174f5dfc5e11bf5bca4dfe6 \ + --hash=sha256:f9f3e9c8a9ecffa57bef8fb4fa19e5fa4d2d8307cf6bac5b1fca5e5860f4ba00 \ + --hash=sha256:fa139875ff98ab97da323cfadfaff08900d1ad42f1b5087b0b812a55c5a06373 \ + --hash=sha256:fcd3b77e2f17bbe4ca56ec7bcb07992647d19d0b9c05d84886dcd6f9eb810afd \ + --hash=sha256:fd8c81f346b58f45818d09ea11db69a8d5fd34a224b79871f6d44f12cd7977b1 \ + --hash=sha256:fe7b7bb170daccbba19ad33012d2b15f1e7942296fd4d45fc1b79013da8cc0f2 \ + --hash=sha256:ff330d3c30db4eb6b01d79e29d2d0b407a7ecad39cfd9ec993ece57396a2ec0d \ + --hash=sha256:ff405d91509d88e8d44129cd87b18d70acd1f0c1aeabd7bc3c46792b1fe2acba \ + --hash=sha256:ffcd54362564dc1a30fb74d8b8a6e5a6b11ebd5e27266adc3b7427a21a6c9104 + # via aiohttp diff --git a/docker/telemetry/workload/rpc_load_generator.py b/docker/telemetry/workload/rpc_load_generator.py index 7b5e22f631..c6b07a6ac5 100644 --- a/docker/telemetry/workload/rpc_load_generator.py +++ b/docker/telemetry/workload/rpc_load_generator.py @@ -30,6 +30,7 @@ import argparse import asyncio import json import logging +import math import random import sys import time @@ -39,6 +40,11 @@ from typing import Any import websockets +# websockets loads its submodules lazily, so websockets.exceptions is not +# reachable through the package until something imports it. REQUEST_FAILURES +# is built at import time and needs it now. +import websockets.exceptions + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -71,9 +77,81 @@ DEFAULT_WEIGHTS: dict[str, int] = { # Well-known genesis account for queries that require an account parameter. GENESIS_ACCOUNT = "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh" +# How long a single request waits for its reply. +RECV_TIMEOUT_S = 10.0 + +# Teardown budget for requests still in flight. Only the at-most-one request +# per connection that already holds the gate can still finish, and its worst +# case is the full receive timeout, so that plus a small grace is the whole +# useful wait. Requests still queued behind the gate would need +# queue_depth x round-trip, which is unbounded; they are cancelled instead. +DRAIN_TIMEOUT_S = RECV_TIMEOUT_S + 2.0 + +# Requests allowed to own one connection's recv() at a time. websockets +# rejects a second concurrent recv() on the same socket, so this must stay 1 +# unless request/response correlation by id is added. Concurrency comes from +# spreading requests round-robin over the endpoints instead. +MAX_INFLIGHT_PER_CONNECTION = 1 + +# Fraction of dispatched requests that must reach the server for a run to +# count as a measurement. The gate is one connection deep, so the ceiling is +# len(connections) / round-trip requests per second; asking for more than +# that silently drops the excess instead of erroring, which would report a +# perfect score on a run that generated a fraction of its intended load. +# Fifty percent mirrors the error-rate limit below, on the same reasoning: a +# run in which most requests did not happen is not a valid baseline. +MIN_DELIVERY_PCT = 50.0 + +# Error rate above which the run is treated as a failure. +MAX_ERROR_RATE_PCT = 50.0 + +# Failures that mean "this request failed", not "the generator is broken": +# asyncio.TimeoutError - no reply within RECV_TIMEOUT_S. Same class as the +# builtin TimeoutError on Python 3.11+. +# WebSocketException - transport failure, including the connection being +# closed while a receive was outstanding, and the +# ConcurrencyError raised for a rejected concurrent +# recv() (it subclasses WebSocketException). +# json.JSONDecodeError - reply body was not valid JSON. +# AttributeError - reply parsed to something with no .get(), e.g. a +# JSON array. +REQUEST_FAILURES: tuple[type[BaseException], ...] = ( + asyncio.TimeoutError, + websockets.exceptions.WebSocketException, + json.JSONDecodeError, + AttributeError, +) + logger = logging.getLogger("rpc_load_generator") +# --------------------------------------------------------------------------- +# Latency helpers +# --------------------------------------------------------------------------- + + +def _percentile(sorted_values: list[float], quantile: float) -> float: + """Return the nearest-rank percentile of an ascending list of values. + + The nearest-rank index is ``ceil(n * q) - 1``, clamped to the last + element. Plain ``int(n * q)`` truncation selects one rank too high + whenever ``n * q`` is a whole number — at n=100 it picks index 99, the + maximum, so the reported p99 was really p100 (n=20 for p95). + + Args: + sorted_values: Values sorted ascending. + quantile: Quantile to select, in (0, 1] — e.g. 0.99. + + Returns: + The selected value, or 0.0 when the list is empty. + """ + n = len(sorted_values) + if n == 0: + return 0.0 + idx = min(math.ceil(n * quantile) - 1, n - 1) + return sorted_values[max(idx, 0)] + + # --------------------------------------------------------------------------- # Data classes # --------------------------------------------------------------------------- @@ -83,55 +161,126 @@ logger = logging.getLogger("rpc_load_generator") class LoadStats: """Tracks request counts and latencies during a load run. + ``total_dispatched`` counts intent and ``total_sent`` counts outcome, so + the difference is the load that never happened. They diverge whenever the + requested rate exceeds what the connections can carry: the dispatch loop + keeps pace, the requests queue behind the per-connection gate, and + teardown cancels whatever never got its turn. Without the two extra + counters that shortfall shows up as nothing at all -- no error, no + warning, and an error_rate_pct of 0 over a fraction of the traffic. + Attributes: - total_sent: Total RPC requests dispatched. - total_success: Requests that returned a valid result. - total_errors: Requests that returned an error or timed out. - latencies: Per-command list of round-trip times in seconds. - command_counts: Per-command request count. + total_dispatched: Requests the dispatch loop created a task for. + total_sent: Requests that completed and were recorded. + total_success: Requests that returned a valid result. + total_errors: Requests that returned an error or timed out. + total_cancelled: Requests cancelled at teardown, never recorded. + latencies: Per-command round-trip times in seconds, for the + requests that got a reply. Requests that never got + one contribute no sample -- see record(). + command_counts: Per-command request count, replied or not. """ + total_dispatched: int = 0 total_sent: int = 0 total_success: int = 0 total_errors: int = 0 + total_cancelled: int = 0 latencies: dict[str, list[float]] = field(default_factory=dict) command_counts: dict[str, int] = field(default_factory=dict) - def record(self, command: str, latency: float, success: bool) -> None: - """Record the outcome of a single RPC call.""" + def record(self, command: str, latency: float | None, success: bool) -> None: + """Record the outcome of a single RPC call. + + Pass ``latency=None`` when no reply arrived, i.e. a timeout or a + transport failure. Such a request still counts as an error, but it + contributes no latency sample: time-to-failure is not a round-trip + time, and a timeout would inject RECV_TIMEOUT_S into the distribution + and dominate the percentiles. + + A reply carrying ``status: error`` is the opposite case. The round + trip completed and was timely, so its latency is a real measurement + and is kept even though the request is counted as an error. + """ self.total_sent += 1 if success: self.total_success += 1 else: self.total_errors += 1 - self.latencies.setdefault(command, []).append(latency) self.command_counts[command] = self.command_counts.get(command, 0) + 1 + if latency is not None: + self.latencies.setdefault(command, []).append(latency) def summary(self) -> dict[str, Any]: - """Return a summary dict suitable for JSON serialization.""" + """Return a summary dict suitable for JSON serialization. + + ``total_sent``, ``total_success``, ``total_errors``, + ``error_rate_pct`` and ``per_command`` keep their names and meanings; + workload_orchestrator.py reads the first and third of those. The three + delivery keys are additions. ``delivery_pct`` is 0.0 when nothing was + dispatched at all -- a run that opened no connection delivered none of + its load, and reporting 100% for it would be the same blind spot the + key exists to close. + """ + # Keyed off command_counts, not latencies: a command whose every + # request timed out has a count but no samples, and dropping it from + # the report would hide the command that failed worst. per_command: dict[str, Any] = {} - for cmd, lats in self.latencies.items(): - sorted_lats = sorted(lats) - n = len(sorted_lats) + for cmd in sorted(self.command_counts): + sorted_lats = sorted(self.latencies.get(cmd, [])) per_command[cmd] = { - "count": self.command_counts.get(cmd, 0), - "p50_ms": round(sorted_lats[n // 2] * 1000, 2) if n else 0, - "p95_ms": (round(sorted_lats[int(n * 0.95)] * 1000, 2) if n else 0), - "p99_ms": (round(sorted_lats[int(n * 0.99)] * 1000, 2) if n else 0), + "count": self.command_counts[cmd], + "latency_samples": len(sorted_lats), + "p50_ms": round(_percentile(sorted_lats, 0.50) * 1000, 2), + "p95_ms": round(_percentile(sorted_lats, 0.95) * 1000, 2), + "p99_ms": round(_percentile(sorted_lats, 0.99) * 1000, 2), } return { + "total_dispatched": self.total_dispatched, "total_sent": self.total_sent, "total_success": self.total_success, "total_errors": self.total_errors, + "total_cancelled": self.total_cancelled, "error_rate_pct": ( round(self.total_errors / self.total_sent * 100, 2) if self.total_sent else 0 ), + "delivery_pct": ( + round(self.total_sent / self.total_dispatched * 100, 2) + if self.total_dispatched + else 0.0 + ), "per_command": per_command, } +@dataclass +class Connection: + """One open WebSocket endpoint together with its request gate. + + websockets raises rather than mis-delivering when two coroutines call + ``recv()`` on the same socket, so every request holds ``gate`` across its + send and its matching receive. Parallelism comes from the round-robin + spread over endpoints: N endpoints allow N requests in flight. + + run_load ──round-robin──> Connection[0] ─gate─> send_rpc (1 at a time) + └─> Connection[1] ─gate─> send_rpc (1 at a time) + + Attributes: + url: WebSocket URL this connection was opened against, for logging. + ws: The open connection. + gate: Limits concurrent send/receive pairs on ``ws`` to + MAX_INFLIGHT_PER_CONNECTION. + """ + + url: str + ws: websockets.ClientConnection + gate: asyncio.Semaphore = field( + default_factory=lambda: asyncio.Semaphore(MAX_INFLIGHT_PER_CONNECTION) + ) + + # --------------------------------------------------------------------------- # RPC command builders # --------------------------------------------------------------------------- @@ -167,7 +316,9 @@ def build_rpc_request(command: str) -> dict[str, Any]: req["limit"] = 5 elif command == "tx": # Use a dummy hash — returns "txnNotFound" error but still exercises - # the full RPC span pipeline (rpc.ws_message -> rpc.process -> rpc.command.tx). + # the full RPC span pipeline for this transport (rpc.ws_message -> + # rpc.command.tx). rpc.process is not in that chain: it is created + # only on the HTTP/JSON-RPC path, which this client never uses. req["transaction"] = "0" * 64 req["binary"] = False elif command == "account_tx": @@ -220,15 +371,23 @@ def choose_command(weights: dict[str, int]) -> str: async def send_rpc( - ws: websockets.WebSocketClientProtocol, + conn: Connection, command: str, stats: LoadStats, inject_traceparent: bool = True, ) -> None: """Send a single RPC request over WebSocket and record the result. + Holds ``conn.gate`` across the send and the matching receive so only one + request at a time owns the connection's ``recv()``. The latency clock + starts after the gate is acquired, so time spent queued behind a busy + connection is not charged to the server. + + Every outcome is recorded, including failures, so error_rate_pct in the + summary reflects every request that was actually sent. + Args: - ws: Open WebSocket connection. + conn: Target connection and its request gate. command: RPC command name. stats: LoadStats instance to record results. inject_traceparent: If True, add a W3C traceparent header field @@ -238,26 +397,174 @@ async def send_rpc( # Inject W3C traceparent for context propagation testing. # The rippled WebSocket handler extracts this from the JSON body - # when present (Phase 2 context propagation). + # when present. if inject_traceparent: trace_id = uuid.uuid4().hex span_id = uuid.uuid4().hex[:16] request["traceparent"] = f"00-{trace_id}-{span_id}-01" - t0 = time.monotonic() - try: - await ws.send(json.dumps(request)) - raw = await asyncio.wait_for(ws.recv(), timeout=10.0) - latency = time.monotonic() - t0 - response = json.loads(raw) - # Native WS responses have {"status": "success", "result": {...}} - # or {"status": "error", "error": "...", "error_message": "..."}. - success = response.get("status") == "success" + async with conn.gate: + t0 = time.monotonic() + # The try covers the I/O and the parse only. Recording sits outside it + # so a bug in record() surfaces as the task failure it is, instead of + # being counted as one more failed request. + try: + await conn.ws.send(json.dumps(request)) + raw = await asyncio.wait_for(conn.ws.recv(), timeout=RECV_TIMEOUT_S) + latency = time.monotonic() - t0 + # Native WS responses have {"status": "success", "result": {...}} + # or {"status": "error", "error": "...", "error_message": "..."}. + success = json.loads(raw).get("status") == "success" + except REQUEST_FAILURES as exc: + logger.debug("RPC %s failed: %s", command, exc) + # No reply, so no latency sample -- see LoadStats.record(). + stats.record(command, None, False) + return stats.record(command, latency, success) - except (asyncio.TimeoutError, websockets.exceptions.WebSocketException) as exc: - latency = time.monotonic() - t0 - stats.record(command, latency, False) - logger.debug("RPC %s failed: %s", command, exc) + + +async def open_connections(endpoints: list[str]) -> list[Connection]: + """Open one persistent WebSocket connection per endpoint. + + Endpoints that refuse the connection are logged and skipped, so a partly + reachable cluster still produces load. + + Args: + endpoints: List of WebSocket URLs (ws://host:port). + + Returns: + The connections that were established, possibly empty. + """ + connections: list[Connection] = [] + for ep in endpoints: + try: + ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10) + connections.append(Connection(url=ep, ws=ws)) + logger.info("Connected to %s", ep) + except Exception as exc: + logger.error("Failed to connect to %s: %s", ep, exc) + return connections + + +async def drain_requests(inflight: set[asyncio.Task[None]]) -> int: + """Let in-flight requests finish, then cancel whatever is still stuck. + + A request may wait up to RECV_TIMEOUT_S for its reply, so closing the + connections straight away would turn late replies into errors. Anything + still unfinished after DRAIN_TIMEOUT_S is cancelled, and the count is + returned so the caller can put the shortfall in the summary instead of + losing it to a log line. + + Args: + inflight: Tasks still tracked as unfinished. Finished tasks remove + themselves, so this is the outstanding set. + + Returns: + Number of requests cancelled without being recorded. + """ + pending = {task for task in inflight if not task.done()} + if not pending: + return 0 + + logger.info( + "Draining %d in-flight request(s), up to %.0fs...", + len(pending), + DRAIN_TIMEOUT_S, + ) + _, stuck = await asyncio.wait(pending, timeout=DRAIN_TIMEOUT_S) + if not stuck: + logger.info("All in-flight requests completed.") + return 0 + + for task in stuck: + task.cancel() + await asyncio.gather(*stuck, return_exceptions=True) + # Most of these never left the client: they were still waiting for the + # per-connection gate. Up to one per connection had already been sent and + # was waiting on recv() when it was cancelled, so the server may have + # handled it. Either way CancelledError is not a REQUEST_FAILURES member, + # so none of them reached stats.record and none are in total_sent. + logger.warning( + "Cancelled %d request(s) unfinished after %.0fs — not counted in " + "total_sent; see total_cancelled and delivery_pct", + len(stuck), + DRAIN_TIMEOUT_S, + ) + return len(stuck) + + +def log_progress(stats: LoadStats, elapsed: float) -> None: + """Log throughput every 100 recorded requests. + + Args: + stats: Live counters. + elapsed: Seconds since the run started. + """ + if stats.total_sent % 100 != 0 or stats.total_sent == 0: + return + logger.info( + "Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)", + stats.total_sent, + stats.total_errors, + stats.total_sent / elapsed if elapsed > 0 else 0, + elapsed, + ) + + +async def dispatch_requests( + connections: list[Connection], + rate: float, + duration: float, + weights: dict[str, int], + stats: LoadStats, + inject_traceparent: bool, +) -> None: + """Fire requests round-robin at the target rate, then drain them. + + Each request runs as its own task so the dispatch loop keeps its pace + regardless of reply latency. Tasks are tracked, not forgotten, so + teardown can drain them and so an exception can never escape unseen. + + Every task created counts towards ``stats.total_dispatched`` and every one + cancelled at teardown towards ``stats.total_cancelled``, which is what + makes an under-delivering run visible in the summary. + + Args: + connections: Open connections to spread requests over. + rate: Target requests per second. + duration: Total run time in seconds. + weights: Command distribution weights. + stats: LoadStats instance to record results in. + inject_traceparent: Whether to inject W3C traceparent headers. + """ + interval = 1.0 / rate if rate > 0 else 0.1 + start = time.monotonic() + conn_idx = 0 + inflight: set[asyncio.Task[None]] = set() + + def reap(task: asyncio.Task[None]) -> None: + """Untrack a finished request and report anything that escaped it.""" + inflight.discard(task) + if not task.cancelled() and task.exception() is not None: + logger.error("RPC task failed unexpectedly: %s", task.exception()) + + try: + while (time.monotonic() - start) < duration: + conn = connections[conn_idx % len(connections)] + conn_idx += 1 + task = asyncio.create_task( + send_rpc(conn, choose_command(weights), stats, inject_traceparent) + ) + inflight.add(task) + task.add_done_callback(reap) + stats.total_dispatched += 1 + + await asyncio.sleep(interval) + log_progress(stats, time.monotonic() - start) + except asyncio.CancelledError: + logger.info("Load generation cancelled.") + finally: + stats.total_cancelled += await drain_requests(inflight) async def run_load( @@ -270,7 +577,9 @@ async def run_load( """Run the RPC load generator against the given endpoints. Distributes requests round-robin across endpoints at the specified - rate (requests per second) for the given duration. + rate (requests per second) for the given duration. Each connection + serves one request at a time, so the ceiling per connection is + 1 / round-trip-latency requests per second; add endpoints to raise it. Args: endpoints: List of WebSocket URLs (ws://host:port). @@ -283,18 +592,8 @@ async def run_load( LoadStats with aggregated results. """ stats = LoadStats() - interval = 1.0 / rate if rate > 0 else 0.1 - - # Open persistent connections to all endpoints. - connections: list[websockets.WebSocketClientProtocol] = [] - for ep in endpoints: - try: - ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10) - connections.append(ws) - logger.info("Connected to %s", ep) - except Exception as exc: - logger.error("Failed to connect to %s: %s", ep, exc) + connections = await open_connections(endpoints) if not connections: logger.error("No connections established. Aborting.") return stats @@ -307,43 +606,26 @@ async def run_load( ) start = time.monotonic() - conn_idx = 0 - try: - while (time.monotonic() - start) < duration: - command = choose_command(weights) - ws = connections[conn_idx % len(connections)] - conn_idx += 1 - - # Fire-and-forget style with bounded concurrency via sleep. - asyncio.create_task(send_rpc(ws, command, stats, inject_traceparent)) - await asyncio.sleep(interval) - - # Periodic progress log. - elapsed = time.monotonic() - start - if stats.total_sent % 100 == 0 and stats.total_sent > 0: - actual_rps = stats.total_sent / elapsed if elapsed > 0 else 0 - logger.info( - "Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)", - stats.total_sent, - stats.total_errors, - actual_rps, - elapsed, - ) - except asyncio.CancelledError: - logger.info("Load generation cancelled.") + await dispatch_requests( + connections, rate, duration, weights, stats, inject_traceparent + ) finally: - # Allow in-flight requests to complete. - await asyncio.sleep(2) - for ws in connections: - await ws.close() + # Only reached once dispatch_requests has drained: closing a + # connection under an outstanding receive raises ConnectionClosed, + # which would be recorded as a failed request. + for conn in connections: + await conn.ws.close() elapsed = time.monotonic() - start logger.info( - "Load complete: %d sent, %d success, %d errors in %.1fs (%.1f RPS)", + "Load complete: %d of %d dispatched sent, %d success, %d errors, " + "%d cancelled in %.1fs (%.1f RPS)", stats.total_sent, + stats.total_dispatched, stats.total_success, stats.total_errors, + stats.total_cancelled, elapsed, stats.total_sent / elapsed if elapsed > 0 else 0, ) @@ -455,9 +737,38 @@ def main() -> None: json.dump(summary, f, indent=2) logger.info("Summary written to %s", args.output) - # Exit with error if error rate exceeds 50%. - if summary["error_rate_pct"] > 50: - logger.error("High error rate: %.1f%%", summary["error_rate_pct"]) + # Both gates are evaluated and reported before exiting, and the summary is + # already on disk, so the caller sees every reason plus the numbers behind + # it. A run that under-delivers has to fail as loudly as one that errors: + # every downstream span and metric assertion would otherwise be checked + # against a fraction of the intended traffic and still look healthy. + failures: list[str] = [] + if summary["error_rate_pct"] > MAX_ERROR_RATE_PCT: + failures.append( + "error rate %.2f%% exceeds %.0f%% (%d of %d requests failed)" + % ( + summary["error_rate_pct"], + MAX_ERROR_RATE_PCT, + summary["total_errors"], + summary["total_sent"], + ) + ) + if summary["delivery_pct"] < MIN_DELIVERY_PCT: + failures.append( + "delivered %.2f%% of dispatched requests, below %.0f%% " + "(%d sent, %d cancelled, of %d dispatched) — lower --rate or add " + "endpoints" + % ( + summary["delivery_pct"], + MIN_DELIVERY_PCT, + summary["total_sent"], + summary["total_cancelled"], + summary["total_dispatched"], + ) + ) + for reason in failures: + logger.error("%s", reason) + if failures: sys.exit(1) diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh index c6c292410b..0cc546ad38 100755 --- a/docker/telemetry/workload/run-full-validation.sh +++ b/docker/telemetry/workload/run-full-validation.sh @@ -19,7 +19,14 @@ # Exit codes: # 0 — All validation checks and the regression gate passed # 1 — Validation checks failed OR the regression gate detected a regression -# 2 — Infrastructure error (cluster/stack failed to start, timing capture failed) +# OR the benchmark exceeded its overhead thresholds +# 2 — Infrastructure error (cluster/stack failed to start, workload +# orchestration failed, timing capture failed, overhead could not be +# measured) +# +# Every step below records its status and folds it into FINAL_EXIT; the first +# non-zero status in pipeline order is the one returned, so the earliest +# failure — the one that explains the later ones — is what the caller sees. set -euo pipefail @@ -35,6 +42,17 @@ die() { exit 2 } +# Overall run status, folded step by step (see the exit-code table above). +FINAL_EXIT=0 + +# fold_exit STATUS — record a step's status in FINAL_EXIT. +# First non-zero wins, so FINAL_EXIT names the earliest failing step. +fold_exit() { + if [ "$1" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then + FINAL_EXIT="$1" + fi +} + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -49,6 +67,9 @@ NUM_NODES=5 RPC_PORT_BASE=5005 WS_PORT_BASE=6006 PEER_PORT_BASE=51235 +# Inert: parsed from --rpc-rate/--rpc-duration/--tx-tps/--tx-duration and never +# read again. Load shape comes from the workload profile instead. Kept because +# the CI workflow still passes the four flags. RPC_RATE=50 RPC_DURATION=120 TX_TPS=5 @@ -78,16 +99,22 @@ usage() { echo "Options:" echo " --xrpld PATH Path to xrpld binary" echo " --nodes NUM Number of validator nodes (default: 5)" - echo " --rpc-rate RPS RPC load rate (default: 50)" - echo " --rpc-duration SECS RPC load duration (default: 120)" - echo " --tx-tps TPS Transaction submit rate (default: 5)" - echo " --tx-duration SECS Transaction submit duration (default: 120)" echo " --profile NAME Workload profile (default: full-validation)" echo " --with-benchmark Also run performance overhead benchmark (telemetry off vs on)" echo " --skip-loki Skip Loki log-trace correlation checks" echo " --skip-regression Skip the OTel-baseline regression gate" echo " --cleanup Tear down everything and exit" echo " -h, --help Show this help" + echo "" + echo "Accepted but INERT (parsed for compatibility, then ignored):" + echo " --rpc-rate RPS no effect" + echo " --rpc-duration SECS no effect" + echo " --tx-tps TPS no effect" + echo " --tx-duration SECS no effect" + echo "" + echo " Load shape comes from the workload profile (--profile), which sets" + echo " the rate and duration of every phase in workload-profiles.json." + echo " These four flags stay accepted because the CI workflow passes them." exit 0 } @@ -101,6 +128,7 @@ while [ $# -gt 0 ]; do NUM_NODES="$2" shift 2 ;; + # The next four are inert — see the RPC_RATE default above. --rpc-rate) RPC_RATE="$2" shift 2 @@ -135,7 +163,10 @@ while [ $# -gt 0 ]; do ;; --cleanup) # Cleanup mode log "Cleaning up..." - pkill -f "$WORKDIR" 2>/dev/null || true + # Match the node config path, not the bare workdir: a plain + # "$WORKDIR" pattern also matches any shell, editor or log tail + # whose command line merely mentions that path. + pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true rm -rf "$WORKDIR" ok "Cleanup complete." @@ -170,7 +201,8 @@ ok "Prerequisites verified." # Cleanup previous run # --------------------------------------------------------------------------- log "Cleaning up previous run..." -pkill -f "$WORKDIR" 2>/dev/null || true +# Narrowed for the same reason as the --cleanup branch above. +pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true sleep 2 rm -rf "$WORKDIR" mkdir -p "$WORKDIR" "$REPORT_DIR" @@ -286,7 +318,6 @@ ${IPS_FIXED} enabled=1 service_instance_id=validator-${i} endpoint=http://localhost:4318/v1/traces -exporter=otlp_http batch_size=512 batch_delay_ms=2000 max_queue_size=2048 @@ -323,9 +354,55 @@ done # --------------------------------------------------------------------------- # Step 3: Wait for consensus # --------------------------------------------------------------------------- +# Report whether a node process is still alive. +# +# A child that has exited but not yet been waited on still answers `kill -0`, +# because the zombie keeps its pid until someone collects it. Checking only +# `kill -0` therefore reads a dead node as alive for the whole readiness +# window, which is how a crashed node came to look like a slow one. +node_running() { + local pid="$1" state + kill -0 "$pid" 2>/dev/null || return 1 + if [ -r "/proc/$pid/stat" ]; then + state=$(awk '{print $3}' "/proc/$pid/stat" 2>/dev/null || echo "?") + [ "$state" != "Z" ] || return 1 + fi + return 0 +} + +# Print why each stopped node stopped: its wait status, then its last output. +# +# The status is the discriminator this harness was missing -- 137 is SIGKILL +# (the kernel reclaiming memory), 139 a segfault, 134 an abort, anything under +# 128 a deliberate exit. The nodes are direct children of this script, so their +# status is still retrievable until something waits on them. +# +# stdout is printed inline rather than left to the artifact upload because a +# node that dies before its debug log opens writes nothing else, and a +# cancelled run uploads nothing at all. +report_stopped_nodes() { + local i pid status + for i in $(seq 1 "$NUM_NODES"); do + pid=$(cat "$WORKDIR/node$i/xrpld.pid" 2>/dev/null || echo "") + [ -n "$pid" ] || continue + node_running "$pid" && continue + status=0 + wait "$pid" 2>/dev/null || status=$? + warn "node$i (pid $pid) is not running — wait status $status" + if [ -s "$WORKDIR/node$i/stdout.log" ]; then + warn "node$i last output:" + tail -n 15 "$WORKDIR/node$i/stdout.log" | sed 's/^/ /' >&2 + else + warn "node$i wrote no stdout at all" + fi + done +} + log "Step 3: Waiting for consensus..." for attempt in $(seq 1 120); do ready=0 + # Reset each attempt so a timeout reports the final state, not a history. + laggards="" for i in $(seq 1 "$NUM_NODES"); do port=$((RPC_PORT_BASE + i - 1)) state=$(curl -sf "http://localhost:$port" \ @@ -333,14 +410,45 @@ for attempt in $(seq 1 120); do jq -r '.result.info.server_state' 2>/dev/null || echo "") if [ "$state" = "proposing" ]; then ready=$((ready + 1)) + else + # Name the node and what it last reported. A bare count says a + # node is missing but not which one, which leaves nothing to grep + # for in the artifacts. An empty state means the RPC port did not + # answer at all, which usually means the process is gone. + laggards="$laggards node$i=${state:-unreachable}" fi done if [ "$ready" -ge "$NUM_NODES" ]; then ok "All $NUM_NODES nodes proposing (attempt $attempt)" break fi + # A stopped process will never reach proposing. Waiting out the rest of the + # window only delays the same failure and buries its cause under two + # minutes of progress output. + stopped=0 + for n in $(seq 1 "$NUM_NODES"); do + p=$(cat "$WORKDIR/node$n/xrpld.pid" 2>/dev/null || echo "") + if [ -n "$p" ] && ! node_running "$p"; then + stopped=$((stopped + 1)) + fi + done + if [ "$stopped" -gt 0 ]; then + echo "" + report_stopped_nodes + die "$stopped of $NUM_NODES node(s) stopped during startup; only $ready reached proposing. Not proposing:${laggards}. Per-node status is above, then '$0 --cleanup'." + fi if [ "$attempt" -eq 120 ]; then - warn "Consensus timeout — $ready/$NUM_NODES nodes ready" + # Fatal, not a warning. A partial cluster still answers queries, so the + # run would complete and report unrelated span/metric failures: series + # counts scale with the number of live nodes, and spans that need a + # quorum are simply never emitted. One infrastructure error here is + # worth more than a pile of misleading assertion failures later. + echo "" + # Every node is still running but not proposing, so this is a genuine + # convergence problem rather than a crash. Run the reporter anyway: it + # is a no-op when nothing stopped, and it costs nothing to be sure. + report_stopped_nodes + die "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${attempt}s. Not proposing:${laggards}. Check $WORKDIR/node*/debug.log and $WORKDIR/node*/stdout.log (a node that died before its log sink opened writes only the latter), then '$0 --cleanup'." fi printf "\r %d/%d nodes proposing..." "$ready" "$NUM_NODES" sleep 1 @@ -357,7 +465,13 @@ for attempt in $(seq 1 60); do ok "Validated ledger: seq $val_seq" break fi - [ "$attempt" -eq 60 ] && warn "No validated ledger after 60s" + # Fatal for the same reason as the consensus timeout above, and because + # several assertions are gated on a validated ledger existing at all: + # ledger_economy{metric="base_fee_xrp"} is only observed from a validated + # ledger, and complete_ledgers stays absent while the range is empty. + if [ "$attempt" -eq 60 ]; then + die "No validated ledger after ${attempt}s (last seq: $val_seq). Check $WORKDIR/node*/debug.log, then '$0 --cleanup'." + fi sleep 1 done @@ -371,14 +485,21 @@ for i in $(seq 1 "$NUM_NODES"); do WS_ENDPOINTS="$WS_ENDPOINTS ws://localhost:$((WS_PORT_BASE + i - 1))" done +ORCHESTRATOR_EXIT=0 python3 "$SCRIPT_DIR/workload_orchestrator.py" \ --profile "$WORKLOAD_PROFILE" \ --endpoints $WS_ENDPOINTS \ --report "$REPORT_DIR/workload-report.json" \ - --report-dir "$REPORT_DIR" || - warn "Workload orchestrator returned non-zero exit" + --report-dir "$REPORT_DIR" || ORCHESTRATOR_EXIT=$? -ok "Workload orchestration complete." +if [ "$ORCHESTRATOR_EXIT" -eq 0 ]; then + ok "Workload orchestration complete." +else + # Treated as an infrastructure error: the span and metric assertions below + # would be graded against traffic that was never generated. + fail "Workload orchestrator failed (exit $ORCHESTRATOR_EXIT) — the checks below run against incomplete traffic" + fold_exit 2 +fi # --------------------------------------------------------------------------- # Step 5: Run telemetry validation suite @@ -398,6 +519,7 @@ if [ "$VALIDATION_EXIT" -eq 0 ]; then else fail "Some telemetry validation checks failed (exit $VALIDATION_EXIT)" fi +fold_exit "$VALIDATION_EXIT" # --------------------------------------------------------------------------- # Step 6: Capture OTel timings and run the regression comparison @@ -444,18 +566,33 @@ if [ "$SKIP_REGRESSION" != true ]; then else warn "Regression gate skipped." fi +fold_exit "$REGRESSION_EXIT" # --------------------------------------------------------------------------- # Step 7: (Optional) Run overhead benchmark # --------------------------------------------------------------------------- +BENCHMARK_EXIT=0 if [ "$WITH_BENCHMARK" = true ]; then log "Step 7: Running performance benchmark..." bash "$SCRIPT_DIR/benchmark.sh" \ --xrpld "$XRPLD" \ --duration 120 \ --nodes 3 \ - --output "$REPORT_DIR" || - warn "Benchmark returned non-zero exit" + --output "$REPORT_DIR" || BENCHMARK_EXIT=$? + + if [ "$BENCHMARK_EXIT" -eq 0 ]; then + ok "Benchmark within overhead thresholds." + elif [ "$BENCHMARK_EXIT" -eq 1 ]; then + # A measured threshold breach — same class as a failed check. + fail "Benchmark exceeded overhead thresholds (exit 1)" + fold_exit 1 + else + # benchmark.sh could not produce a usable measurement (e.g. incomplete + # system metrics). Reported as an infrastructure error, not a perf + # regression: nothing was measured, so nothing was breached. + fail "Benchmark could not measure overhead (exit $BENCHMARK_EXIT) — treated as an infrastructure error" + fold_exit 2 + fi fi # --------------------------------------------------------------------------- @@ -486,15 +623,13 @@ echo "" echo " To tear down:" echo " $0 --cleanup" echo "" +echo " Step statuses (0 = ok):" +echo " Workload orchestration: $ORCHESTRATOR_EXIT" +echo " Telemetry validation: $VALIDATION_EXIT" +echo " Regression gate: $REGRESSION_EXIT" +echo " Overhead benchmark: $BENCHMARK_EXIT" +echo "" echo "===========================================================" -# Fail the run if EITHER validation or the regression gate failed. The -# `[ "$VAR" -gt N ]` comparison works here because exit codes are numeric. -FINAL_EXIT=0 -if [ "$VALIDATION_EXIT" -ne 0 ]; then - FINAL_EXIT="$VALIDATION_EXIT" -fi -if [ "$REGRESSION_EXIT" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then - FINAL_EXIT="$REGRESSION_EXIT" -fi +# FINAL_EXIT already holds the first non-zero step status (see fold_exit). exit "$FINAL_EXIT" diff --git a/docker/telemetry/workload/tx_submitter.py b/docker/telemetry/workload/tx_submitter.py index 4d779ec98e..745ce0ae9a 100644 --- a/docker/telemetry/workload/tx_submitter.py +++ b/docker/telemetry/workload/tx_submitter.py @@ -90,6 +90,57 @@ DEFAULT_TX_WEIGHTS: dict[str, int] = { # Number of test accounts to create. NUM_TEST_ACCOUNTS = 8 +# Minimum number of funded accounts the transaction builders need: TX_BUILDERS +# indexes positions 0..5 of the account list. +MIN_FUNDED_ACCOUNTS = 6 + +# Engine results that tie up the submitted sequence number, other than the +# tec* family which is matched by prefix. See consumes_sequence(). +SEQ_CONSUMING_RESULTS = frozenset({"tesSUCCESS", "terQUEUED"}) + +# Consecutive non-consuming submit results from one account before its +# sequence is re-read from the ledger. The periodic refresh only ever raises +# the counter, so a counter that leads the ledger -- a queued transaction that +# was later dropped, say -- would otherwise make every further submit from +# that account fail forever. Five is above the two or three rejections a +# single bad transaction type can produce in a row, and at the default 5 TPS +# it triggers within a few seconds, well before the periodic refresh below. +SEQ_REFETCH_AFTER_FAILURES = 5 + +# How often the submission loop re-reads every account's sequence from the +# ledger, to stay close to sequences other submitters have advanced. +SEQ_REFRESH_INTERVAL_S = 10.0 + + +def consumes_sequence(engine_result: str | None) -> bool: + """Report whether an engine result tied up the submitted sequence number. + + Two outcomes do. ``tesSUCCESS`` is applied (TER.h:243) and every ``tec*`` + claims the fee "to use the sequence number" (TER.h:265-266), so both + advance the account root. ``terQUEUED`` is the one ``ter*`` code rippled + forwards (TER.h:205); the transaction waits in the queue still holding + that sequence, so the next submit needs the following one. + + Everything else leaves the sequence free. ``tem*``, ``tef*`` and ``tel*`` + are neither applied nor forwarded, and neither are the remaining ``ter*`` + codes (TER.h:202-206) -- including ``terPRE_SEQ``, which reports that the + submitted sequence is already past the account root (TER.h:217). + Advancing the local counter on those widens the gap TER.h:209 calls a + "hole in sequence which jams transactions", and it is what + SEQ_REFETCH_AFTER_FAILURES exists to recover from. + + Args: + engine_result: The ``engine_result`` field of a submit response. A + missing or JSON-null field reads as non-consuming + rather than raising, so the caller records the + transaction exactly once. + + Returns: + True if the submitted sequence number is spoken for. + """ + result = str(engine_result or "") + return result in SEQ_CONSUMING_RESULTS or result.startswith("tec") + # --------------------------------------------------------------------------- # Data classes @@ -98,19 +149,28 @@ NUM_TEST_ACCOUNTS = 8 @dataclass class Account: - """Represents a funded XRPL test account. + """Represents an XRPL test account, funded or not. Attributes: name: Human-readable name (e.g., "alice"). account: Classic address (rXXX...). seed: Secret seed for signing. sequence: Next available sequence number. + funded: True once the account root exists on the ledger. False + accounts cannot submit, so they are excluded from the + submission loop rather than silently counted. + stalled: Consecutive submit results from this account that consumed + no sequence. Reset by any consuming result; at + SEQ_REFETCH_AFTER_FAILURES the sequence is re-read from + the ledger. """ name: str account: str seed: str sequence: int = 0 + funded: bool = False + stalled: int = 0 @dataclass @@ -163,7 +223,7 @@ class TxStats: async def ws_request( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, command: str, params: dict[str, Any] | None = None, ) -> dict[str, Any]: @@ -204,7 +264,7 @@ async def ws_request( return resp.get("result", resp) -async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> Account: +async def create_account(ws: websockets.ClientConnection, name: str) -> Account: """Create a new account via wallet_propose RPC. Args: @@ -227,7 +287,7 @@ async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> A async def fund_account( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, dest: Account, genesis_seq: int, ) -> tuple[bool, int]: @@ -239,7 +299,8 @@ async def fund_account( genesis_seq: Current genesis account sequence number. Returns: - Tuple of (success: bool, next_sequence: int). + Tuple of (funded: bool, next_genesis_sequence: int). The sequence is + unchanged when the ledger did not consume it. """ resp = await ws_request( ws, @@ -265,12 +326,16 @@ async def fund_account( engine_result, json.dumps(resp, indent=None)[:500], ) - return success, genesis_seq + 1 + # Advance the genesis sequence only when the ledger consumed it. The + # caller reads it once and threads it through every funding submit, so + # advancing past a tem*/tef*/tel* rejection would put every remaining + # submit on a future sequence and fund nothing. + if consumes_sequence(engine_result): + genesis_seq += 1 + return success, genesis_seq -async def get_account_sequence( - ws: websockets.WebSocketClientProtocol, account: str -) -> int: +async def get_account_sequence(ws: websockets.ClientConnection, account: str) -> int: """Get the current sequence number for an account. Args: @@ -564,7 +629,7 @@ TX_BUILDERS: dict[str, Any] = { async def setup_accounts( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, ) -> list[Account]: """Create and fund test accounts from genesis. @@ -575,7 +640,8 @@ async def setup_accounts( ws: Open WebSocket connection to a rippled node. Returns: - List of funded Account instances. + Every created Account. ``funded`` marks the ones the ledger accepted, + so the caller must filter on it rather than on the list length. """ account_names = ["alice", "bob", "carol", "dave", "eve", "frank", "grace", "heidi"] @@ -593,8 +659,8 @@ async def setup_accounts( # Fund all accounts. logger.info("Funding test accounts...") for acct in accounts: - success, genesis_seq = await fund_account(ws, acct, genesis_seq) - if success: + acct.funded, genesis_seq = await fund_account(ws, acct, genesis_seq) + if acct.funded: logger.info(" Funded %s", acct.name) else: logger.warning(" Failed to fund %s", acct.name) @@ -603,19 +669,34 @@ async def setup_accounts( logger.info("Waiting 10s for funding transactions to validate...") await asyncio.sleep(10) - # Refresh sequence numbers for all accounts. + # Refresh sequence numbers, and confirm funding against the ledger rather + # than trusting the submit result. get_account_sequence returns 0 when + # account_info reports no account_data, which means the account root was + # never created; a sequence we cannot read also makes the account + # unusable, so either way it must not be submitted from. for acct in accounts: try: acct.sequence = await get_account_sequence(ws, acct.account) - logger.info(" %s sequence: %d", acct.name, acct.sequence) except Exception as exc: logger.warning(" Failed to get sequence for %s: %s", acct.name, exc) + if acct.sequence > 0: + logger.info(" %s sequence: %d", acct.name, acct.sequence) + else: + acct.funded = False + logger.warning( + " %s has no ledger sequence — treating as unfunded", acct.name + ) + logger.info( + "Funded %d of %d created accounts", + sum(1 for a in accounts if a.funded), + len(accounts), + ) return accounts async def submit_transaction( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, tx_type: str, accounts: list[Account], stats: TxStats, @@ -652,8 +733,13 @@ async def submit_transaction( ) stats.record(tx_type, success) + # The sequence gate is deliberately not `success`: that tuple is + # narrower (every tec* consumes a sequence, only two are listed) and + # wider (a tem*/tef*/tel*/ter* rejection consumes none) than the set + # of results that actually consume one. _track_sequence also owns the + # recovery path for a counter that has drifted ahead of the ledger. if sender: - sender.sequence += 1 + await _track_sequence(ws, sender, engine_result) if not success: # First occurrence of each distinct result at WARNING, the rest at @@ -675,15 +761,77 @@ async def submit_transaction( _log_first_failure("exc:%s" % type(exc).__name__, "%s error: %s", tx_type, exc) +async def _track_sequence( + ws: websockets.ClientConnection, + sender: Account, + engine_result: str | None, +) -> None: + """Advance, or re-read from the ledger, one sender's sequence number. + + A consuming result moves the counter on by one and clears the stall + streak. A non-consuming one leaves the counter where it is, so the same + sequence is offered again -- correct when the rejection was about the + transaction, but a livelock when the counter itself is the problem, since + _refresh_sequences never lowers it. After SEQ_REFETCH_AFTER_FAILURES + consecutive non-consuming results the ledger's own value is taken + instead, in either direction. + + A zero from get_account_sequence means account_info returned no + account_data, so it is ignored rather than written over a usable counter. + + Args: + ws: Open WebSocket connection. + sender: The account the transaction was submitted from. + engine_result: The ``engine_result`` of that submit. + """ + if consumes_sequence(engine_result): + sender.sequence += 1 + sender.stalled = 0 + return + + sender.stalled += 1 + if sender.stalled < SEQ_REFETCH_AFTER_FAILURES: + return + + sender.stalled = 0 + try: + seq = await get_account_sequence(ws, sender.account) + except Exception as exc: + logger.warning("Sequence re-fetch for %s failed: %s", sender.name, exc) + return + if seq > 0 and seq != sender.sequence: + logger.warning( + "Re-syncing %s sequence %d -> %d after %d non-consuming results", + sender.name, + sender.sequence, + seq, + SEQ_REFETCH_AFTER_FAILURES, + ) + sender.sequence = seq + + async def _refresh_sequences( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, accounts: list[Account], ) -> None: """Re-sync account sequences from the validated ledger. In a consensus network, other nodes' transactions advance sequences - beyond the submitter's local tracking. Refreshing every ~10 s keeps - the local counter close to the ledger and prevents tefPAST_SEQ storms. + beyond the submitter's local tracking. Refreshing every + SEQ_REFRESH_INTERVAL_S keeps the local counter close to the ledger and + prevents tefPAST_SEQ storms. + + The counter is only ever raised here, never lowered, because it may + legitimately lead the ledger: a queued transaction holds its sequence + without having applied yet, and lowering the counter would reuse it. + Raising it fixes the opposite case, where the ledger has moved on -- a + submit response that was lost while the transaction applied, or another + submitter using the same account. + + That leaves one case this cannot fix: a counter that leads the ledger and + never catches up, because the transaction it was advanced for was dropped + rather than applied. _track_sequence handles that one by re-reading the + ledger value in either direction. """ for acct in accounts: try: @@ -694,6 +842,59 @@ async def _refresh_sequences( pass +async def _submission_loop( + ws: websockets.ClientConnection, + accounts: list[Account], + weights: dict[str, int], + duration: float, + interval: float, + stats: TxStats, +) -> float: + """Submit a weighted transaction mix until ``duration`` elapses. + + Args: + ws: Open WebSocket connection. + accounts: Funded accounts to submit from. + weights: Transaction type distribution weights. + duration: Run time in seconds. + interval: Delay between submissions, i.e. 1 / target TPS. + stats: TxStats instance to record results in. + + Returns: + Seconds actually spent in the loop. + """ + tx_types = list(weights.keys()) + tx_weights = [weights[t] for t in tx_types] + + start = time.monotonic() + last_seq_refresh = start + while (time.monotonic() - start) < duration: + # Periodically re-sync account sequences from the ledger so + # locally-tracked sequences don't drift behind consensus. + if (time.monotonic() - last_seq_refresh) >= SEQ_REFRESH_INTERVAL_S: + await _refresh_sequences(ws, accounts) + last_seq_refresh = time.monotonic() + + tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0] + await submit_transaction(ws, tx_type, accounts, stats) + await asyncio.sleep(interval) + + # Progress logging every 50 transactions. + if stats.total_submitted % 50 == 0 and stats.total_submitted > 0: + elapsed = time.monotonic() - start + logger.info( + "Progress: %d submitted, %d success, %d errors, " + "%.1f TPS (%.0fs elapsed)", + stats.total_submitted, + stats.total_success, + stats.total_errors, + stats.total_submitted / elapsed if elapsed > 0 else 0, + elapsed, + ) + + return time.monotonic() - start + + async def run_submitter( endpoint: str, tps: float, @@ -713,60 +914,40 @@ async def run_submitter( """ stats = TxStats() interval = 1.0 / tps if tps > 0 else 0.5 + elapsed = 0.0 ws = await websockets.connect(endpoint, ping_interval=20, ping_timeout=10) logger.info("Connected to %s", endpoint) try: - # Setup test accounts. - accounts = await setup_accounts(ws) - if len(accounts) < 6: - logger.error("Need at least 6 funded accounts, got %d", len(accounts)) + # Setup test accounts. Every created account is returned whether or + # not funding worked, so submit only from the funded ones — the + # builders address accounts by position and an unfunded account there + # would fail every transaction it is picked for. + created = await setup_accounts(ws) + accounts = [acct for acct in created if acct.funded] + if len(accounts) < MIN_FUNDED_ACCOUNTS: + logger.error( + "Need at least %d funded accounts, only %d of %d created " + "accounts were funded", + MIN_FUNDED_ACCOUNTS, + len(accounts), + len(created), + ) return stats - # Build weighted command list. - tx_types = list(weights.keys()) - tx_weights = [weights[t] for t in tx_types] - logger.info( "Starting TX submission: tps=%s, duration=%ss, types=%d", tps, duration, - len(tx_types), + len(weights), + ) + elapsed = await _submission_loop( + ws, accounts, weights, duration, interval, stats ) - - start = time.monotonic() - last_seq_refresh = start - seq_refresh_interval = 10.0 - while (time.monotonic() - start) < duration: - # Periodically re-sync account sequences from the ledger so - # locally-tracked sequences don't drift behind consensus. - if (time.monotonic() - last_seq_refresh) >= seq_refresh_interval: - await _refresh_sequences(ws, accounts) - last_seq_refresh = time.monotonic() - - tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0] - await submit_transaction(ws, tx_type, accounts, stats) - await asyncio.sleep(interval) - - # Progress logging every 50 transactions. - if stats.total_submitted % 50 == 0 and stats.total_submitted > 0: - elapsed = time.monotonic() - start - actual_tps = stats.total_submitted / elapsed if elapsed > 0 else 0 - logger.info( - "Progress: %d submitted, %d success, %d errors, " - "%.1f TPS (%.0fs elapsed)", - stats.total_submitted, - stats.total_success, - stats.total_errors, - actual_tps, - elapsed, - ) - finally: await ws.close() - elapsed = time.monotonic() - start logger.info( "Submission complete: %d submitted, %d success, %d errors " "in %.1fs (%.1f TPS)", diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py index a6e1c9524d..0fea028ede 100644 --- a/docker/telemetry/workload/validate_telemetry.py +++ b/docker/telemetry/workload/validate_telemetry.py @@ -6,12 +6,15 @@ a workload run. Queries Tempo (spans), Prometheus (metrics), Loki (logs), and Grafana (dashboards) APIs to produce a pass/fail report. Validation categories: - 1. Span validation — All 16+ span types present with required attributes - 2. Metric validation — SpanMetrics, StatsD, and Phase 9 metrics are non-zero + 1. Span validation — Every required span type in expected_spans.json, each + carrying its required attributes + 2. Metric validation — SpanMetrics, StatsD, and MetricsRegistry OTLP metrics + are non-zero 3. Sync diagnostics — Fresh-node sync signals (bootstrap + acquire pipeline) declared in the "sync_diagnostics" group 4. Log-trace correlation — Loki logs contain trace_id/span_id fields - 5. Dashboard validation — All 15 Grafana dashboards render data + 5. Dashboard validation — Every dashboard uid in expected_metrics.json + provisions and loads (panel count only, not panel data) 6. External parity — Span attrs, metric existence, and value sanity for external dashboard parity (validator-health, peer-quality, node-health) @@ -29,6 +32,7 @@ Usage: import argparse import asyncio +import fnmatch import json import logging import sys @@ -70,6 +74,12 @@ METRIC_POLL_INTERVAL_SEC = 5.0 # category and a single, explicit failure per missing metric. SYNC_DIAGNOSTICS_GROUP = "sync_diagnostics" +# All metrics are polled concurrently against ONE shared deadline, so the +# metric phase costs a single poll window instead of one per metric. This caps +# how many /api/v1/series requests are in flight at a time, so the fan-out does +# not hammer the single-container Prometheus the harness runs. +METRIC_POLL_CONCURRENCY = 8 + # --------------------------------------------------------------------------- # Data classes @@ -226,6 +236,26 @@ def _otlp_span_attr_keys(span: dict[str, Any]) -> set[str]: return {a["key"] for a in span.get("attributes", []) if "key" in a} +def _span_name_matches(emitted_name: str, expected_name: str) -> bool: + """Test an emitted span name against a name from expected_spans.json. + + Contract names are either literals or globs containing "*" (for example + "rpc.command.*"). Literals are compared for exact equality so a longer + emitted name cannot satisfy a shorter contract: "consensus.accept.apply" + must not stand in for "consensus.accept". + + Args: + emitted_name: Span name as reported by Tempo. + expected_name: Span name or glob pattern from expected_spans.json. + + Returns: + True when the emitted name satisfies the expected name. + """ + if "*" in expected_name: + return fnmatch.fnmatchcase(emitted_name, expected_name) + return emitted_name == expected_name + + # --------------------------------------------------------------------------- # Span Validation (Tempo API) # --------------------------------------------------------------------------- @@ -377,10 +407,9 @@ async def validate_spans( # Validate required attributes on first trace. if count > 0 and span_def.get("required_attributes"): - trace_id = traces[0].get("traceID", "") - if trace_id: - spans = await _tempo_get_trace(session, tempo_url, trace_id) - await _validate_span_attributes_otlp(spans, span_def, report) + await _check_attributes_on_first_trace( + session, tempo_url, traces, span_def, report + ) except Exception as exc: report.add( CheckResult( @@ -407,15 +436,66 @@ async def validate_spans( await _validate_parent_child(session, tempo_url, rel, report) +async def _check_attributes_on_first_trace( + session: aiohttp.ClientSession, + tempo_url: str, + traces: list[dict[str, Any]], + span_def: dict[str, Any], + report: ValidationReport, +) -> None: + """Fetch the first trace and check the span's required attributes. + + Fetching the trace is a second network call, so it carries its own error + handling. Letting it fall through to the caller's handler would add a + second result under the span's own check name, which has already recorded + the trace as found -- one entry passing and one failing for the same name, + inflating the check total and blaming the trace-existence check for an + attribute-fetch failure. + + Args: + session: aiohttp client session. + tempo_url: Base URL for the Tempo API. + traces: Traces returned for this span, most recent first. + span_def: The span's entry from expected_spans.json. + report: ValidationReport to accumulate results. + """ + span_name = span_def["name"] + try: + trace_id = traces[0].get("traceID", "") + if not trace_id: + return + spans = await _tempo_get_trace(session, tempo_url, trace_id) + await _validate_span_attributes_otlp(spans, span_def, report) + except Exception as exc: + report.add( + CheckResult( + name=f"span.attrs.{span_name}", + category="span", + passed=False, + message=f"{span_name}: attribute check failed ({exc})", + ) + ) + + async def _validate_span_attributes_otlp( spans: list[dict[str, Any]], span_def: dict[str, Any], report: ValidationReport, ) -> None: - """Check that OTLP spans contain expected attributes. + """Check that the contract's own span carries its required attributes. + + Only spans whose name matches ``span_def["name"]`` are inspected. + Attributes are never borrowed from siblings: many span types share keys + such as ledger_seq or tx_hash, so a trace-wide scan would satisfy every + one of those contracts from a single carrier span and make the per-span + contract unenforceable. + + A span type passes when at least one instance of it carries every required + attribute. When none does, the closest instance's missing keys are + reported. Args: - spans: List of OTLP span dicts from Tempo. + spans: Every OTLP span dict in the fetched trace. span_def: Span definition from expected_spans.json. report: ValidationReport to accumulate results. """ @@ -424,26 +504,53 @@ async def _validate_span_attributes_otlp( return span_name = span_def["name"] - # Collect all attribute keys from all spans. - found_attrs: set[str] = set() - for span in spans: - found_attrs.update(_otlp_span_attr_keys(span)) + check_name = f"span.attrs.{span_name}" + matching = [s for s in spans if _span_name_matches(s.get("name", ""), span_name)] + + if not matching: + report.add( + CheckResult( + name=check_name, + category="span", + passed=False, + message=( + f"{span_name}: no span named '{span_name}' in the fetched " + "trace, cannot verify its attributes" + ), + details={"required": required_attrs, "instances": 0}, + ) + ) + return + + # Keep the instance that is missing the fewest required attributes, so the + # failure message names the closest witness rather than an arbitrary one. + best_found: set[str] = set() + best_missing: list[str] = list(required_attrs) + for span in matching: + found = _otlp_span_attr_keys(span) + missing = [a for a in required_attrs if a not in found] + if len(missing) < len(best_missing): + best_found, best_missing = found, missing + if not best_missing: + break - missing = [a for a in required_attrs if a not in found_attrs] report.add( CheckResult( - name=f"span.attrs.{span_name}", + name=check_name, category="span", - passed=len(missing) == 0, + passed=not best_missing, message=( f"{span_name}: all {len(required_attrs)} attributes present" - if not missing - else f"{span_name}: missing attributes: {missing}" + if not best_missing + else f"{span_name}: no '{span_name}' span carried all " + f"{len(required_attrs)} required attributes; closest of " + f"{len(matching)} instance(s) missing {best_missing}" ), details={ "required": required_attrs, - "found": list(found_attrs), - "missing": missing, + "found": sorted(best_found), + "missing": best_missing, + "instances": len(matching), }, ) ) @@ -482,21 +589,21 @@ async def _validate_parent_child( ) return - # Check if child spans exist within parent traces. - # Use the concrete child name for wildcard patterns. - concrete_child = child_name.replace("*", "server_info") + # Check if child spans exist within parent traces. Names are matched + # exactly (globs for wildcard contracts) — a substring test let a + # longer emitted name satisfy a shorter contract, so + # consensus.round -> consensus.accept passed on a + # consensus.accept.apply span alone. found_child = False for trace_summary in traces: trace_id = trace_summary.get("traceID", "") if not trace_id: continue spans = await _tempo_get_trace(session, tempo_url, trace_id) - for span in spans: - op = span.get("name", "") - if concrete_child in op or ("*" not in child_name and op == child_name): - found_child = True - break - if found_child: + if any( + _span_name_matches(span.get("name", ""), child_name) for span in spans + ): + found_child = True break report.add( @@ -669,29 +776,25 @@ async def _validate_trace_join_group( SKIPPED_METRIC_GROUPS = ("description", "grafana_dashboards", SYNC_DIAGNOSTICS_GROUP) -async def validate_metrics( - session: aiohttp.ClientSession, - prometheus_url: str, - report: ValidationReport, +async def _log_prometheus_metric_names( + session: aiohttp.ClientSession, prometheus_url: str ) -> None: - """Validate that expected metrics appear in Prometheus with non-zero values. + """Log the harness-relevant metric names Prometheus currently knows. + + Diagnostic only — this output appears in CI logs and helps debug name + mismatches between expected_metrics.json and actual emissions. Failures + are warnings, never check failures. Args: session: aiohttp client session. - prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090). - report: ValidationReport to accumulate results. + prometheus_url: Prometheus base URL. """ - logger.info("--- Metric Validation (Prometheus) ---") - - # Diagnostic: list all metric names in Prometheus. Helps debug name - # mismatches between expected_metrics.json and actual emissions. try: async with session.get( f"{prometheus_url}/api/v1/label/__name__/values" ) as resp: label_data = await resp.json() all_metrics = label_data.get("data", []) - # Log relevant metrics for debugging. relevant = [ m for m in all_metrics @@ -763,22 +866,99 @@ async def validate_metrics( except Exception as exc: logger.warning("Failed to fetch Prometheus metric names: %s", exc) + +async def validate_metrics( + session: aiohttp.ClientSession, + prometheus_url: str, + report: ValidationReport, +) -> None: + """Validate that expected metrics appear in Prometheus with non-zero values. + + Args: + session: aiohttp client session. + prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090). + report: ValidationReport to accumulate results. + """ + logger.info("--- Metric Validation (Prometheus) ---") + + await _log_prometheus_metric_names(session, prometheus_url) + with open(EXPECTED_METRICS_FILE) as f: expected = json.load(f) - # Check each metric category. SKIPPED_METRIC_GROUPS keys are either not - # metric groups at all or are owned by a dedicated validator below, so - # skipping them here keeps each group to a single owner (no duplicate - # Prometheus queries and no duplicate report entries). - for category_key, category_data in expected.items(): - if category_key in SKIPPED_METRIC_GROUPS: - continue + # Flatten every (category, metric) pair the contract asserts, then poll + # them concurrently against ONE shared deadline. Polling them serially made + # each metric own its own timeout, so the waits were additive: 58 metrics x + # 45 s = 43.5 min, which overran the CI job budget and lost the + # artifact-upload and summary diagnostics. Sharing the deadline bounds the + # whole phase to a single poll window. + targets = [ + (category_key, metric_name) + for category_key, category_data in expected.items() + if category_key not in SKIPPED_METRIC_GROUPS + for metric_name in category_data.get("metrics", []) + ] - metrics = category_data.get("metrics", []) - for metric_name in metrics: - await _check_prometheus_metric( - session, prometheus_url, metric_name, category_key, report + deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC + sem = asyncio.Semaphore(METRIC_POLL_CONCURRENCY) + checks = await asyncio.gather( + *( + _check_prometheus_metric( + session, prometheus_url, metric_name, category, deadline, sem ) + for category, metric_name in targets + ) + ) + + # Add in contract order, not completion order, so the report and its log + # lines stay deterministic across runs. + for check in checks: + report.add(check) + + +async def _poll_series_count( + session: aiohttp.ClientSession, + prometheus_url: str, + metric_name: str, + deadline: float, + sem: asyncio.Semaphore, +) -> int: + """Poll Prometheus until a metric has series or the deadline passes. + + Uses the /api/v1/series endpoint instead of an instant query. + Beast::insight StatsD gauges only mark dirty on value *changes*, so a gauge + that stabilizes (e.g. peer count stays at 1) may go stale in Prometheus and + disappear from instant queries. The series endpoint returns any metric + that existed in the window, regardless of staleness. + + Polls rather than querying once: late-populating gauges/counters may not + have completed the export+scrape pipeline when this runs, so a single query + races. A metric that never appears still fails once the deadline passes. + + Args: + session: aiohttp client session. + prometheus_url: Prometheus base URL. + metric_name: Prometheus metric name. + deadline: Monotonic deadline shared by every metric in the run. + sem: Bounds how many requests reach Prometheus at once. It + is held only across the request, never across the + sleep, so one absent metric cannot starve the others. + + Returns: + Number of series found, or 0 if the metric never appeared. + """ + params: dict[str, str] = {"match[]": metric_name} + while True: + async with sem: + async with session.get( + f"{prometheus_url}/api/v1/series", params=params + ) as resp: + data = await resp.json() + series_count = len(data.get("data", [])) + if series_count > 0 or time.monotonic() >= deadline: + return series_count + # Never sleep past the shared deadline. + await asyncio.sleep(min(METRIC_POLL_INTERVAL_SEC, deadline - time.monotonic())) async def _check_prometheus_metric( @@ -786,8 +966,9 @@ async def _check_prometheus_metric( prometheus_url: str, metric_name: str, category: str, - report: ValidationReport, -) -> None: + deadline: float, + sem: asyncio.Semaphore, +) -> CheckResult: """Query Prometheus for a specific metric and check it exists. Args: @@ -795,54 +976,34 @@ async def _check_prometheus_metric( prometheus_url: Prometheus base URL. metric_name: Prometheus metric name. category: Metric category for the report. - report: ValidationReport to accumulate results. + deadline: Monotonic deadline shared by every metric in the run. + sem: Bounds how many requests reach Prometheus at once. + + Returns: + The CheckResult for this metric. The caller adds it to the report so + report order follows the contract file rather than completion order. """ try: - # Use the /api/v1/series endpoint instead of an instant query. - # Beast::insight StatsD gauges only mark dirty on value *changes*, - # so a gauge that stabilizes (e.g. peer count stays at 1) may go - # stale in Prometheus and disappear from instant queries. The - # series endpoint returns any metric that existed in the window, - # regardless of staleness. - # - # Poll rather than query once: late-populating gauges/counters may - # not have completed the export+scrape pipeline when this runs, so a - # single query races. Re-query until the metric appears or the poll - # window elapses; a metric that never appears still fails after the - # timeout. - params: dict[str, str] = {"match[]": metric_name} - series_count = 0 - deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC - while True: - async with session.get( - f"{prometheus_url}/api/v1/series", params=params - ) as resp: - data = await resp.json() - series_count = len(data.get("data", [])) - if series_count > 0 or time.monotonic() >= deadline: - break - await asyncio.sleep(METRIC_POLL_INTERVAL_SEC) - report.add( - CheckResult( - name=f"metric.{category}.{metric_name}", - category="metric", - passed=series_count > 0, - message=( - f"{metric_name}: {series_count} series" - if series_count > 0 - else f"{metric_name}: 0 series (expected > 0)" - ), - details={"series_count": series_count}, - ) + series_count = await _poll_series_count( + session, prometheus_url, metric_name, deadline, sem + ) + return CheckResult( + name=f"metric.{category}.{metric_name}", + category="metric", + passed=series_count > 0, + message=( + f"{metric_name}: {series_count} series" + if series_count > 0 + else f"{metric_name}: 0 series (expected > 0)" + ), + details={"series_count": series_count}, ) except Exception as exc: - report.add( - CheckResult( - name=f"metric.{category}.{metric_name}", - category="metric", - passed=False, - message=f"{metric_name}: query failed ({exc})", - ) + return CheckResult( + name=f"metric.{category}.{metric_name}", + category="metric", + passed=False, + message=f"{metric_name}: query failed ({exc})", ) @@ -1032,6 +1193,32 @@ async def validate_log_trace_correlation( # --------------------------------------------------------------------------- +def _leaf_panel_count(dashboard: dict[str, Any]) -> int: + """Count the panels a dashboard actually renders. + + Grafana models a row as an entry of ``type: "row"`` in the top-level + ``panels`` list, and a collapsed row carries its children in its own + nested ``panels`` list. So ``len(dashboard["panels"])`` counts rows as + though they were panels and misses everything inside a collapsed one -- + on ``node-health`` that reads 55 where the true figure is 51, and a + dashboard consisting only of collapsed rows would report a positive + count while rendering nothing. + + Args: + dashboard: The ``dashboard`` object from the Grafana API response. + + Returns: + The number of non-row panels, including those nested inside rows. + """ + total = 0 + for panel in dashboard.get("panels", []): + if panel.get("type") == "row": + total += len(panel.get("panels", [])) + else: + total += 1 + return total + + async def validate_dashboards( session: aiohttp.ClientSession, grafana_url: str, @@ -1060,13 +1247,17 @@ async def validate_dashboards( if resp.status == 200: data = await resp.json() dashboard = data.get("dashboard", {}) - panel_count = len(dashboard.get("panels", [])) + panel_count = _leaf_panel_count(dashboard) report.add( CheckResult( name=f"dashboard.{uid}", category="dashboard", - passed=True, - message=(f"{uid}: loaded ({panel_count} panels)"), + passed=panel_count > 0, + message=( + f"{uid}: loaded ({panel_count} panels)" + if panel_count + else f"{uid}: loaded but renders no panels" + ), details={"panel_count": panel_count}, ) ) @@ -1315,6 +1506,131 @@ async def validate_parity_span_attrs( ) +def _series_label(series: dict[str, Any]) -> str: + """Name a Prometheus series for use in a failure message. + + Args: + series: One entry from a Prometheus query result. + + Returns: + The series' service_instance_id when it carries one (the label that + tells harness cluster nodes apart), else its full label set. + """ + metric = series.get("metric", {}) + instance = metric.get("service_instance_id") + if instance: + return f"service_instance_id={instance}" + return str(metric) if metric else "" + + +def _value_in_bounds( + value: float, lo: float, hi: float | None, exclusive_lo: bool +) -> bool: + """Test one sample against a sanity range. + + Args: + value: Sample value. + lo: Lower bound. + hi: Upper bound, or None when unbounded above. + exclusive_lo: True when the lower bound is exclusive. + + Returns: + True when the value is inside the range. + """ + lo_ok = value > lo if exclusive_lo else value >= lo + return lo_ok and (hi is None or value <= hi) + + +def _bounds_description(lo: float, hi: float | None, exclusive_lo: bool) -> str: + """Build the human-readable bound text used in check messages. + + Args: + lo: Lower bound. + hi: Upper bound, or None when unbounded above. + exclusive_lo: True when the lower bound is exclusive. + + Returns: + A phrase such as "> 0 and <= 100". + """ + desc = f"{'>' if exclusive_lo else '>='} {lo}" + if hi is not None: + desc += f" and <= {hi}" + return desc + + +async def _check_parity_value( + session: aiohttp.ClientSession, + prometheus_url: str, + entry: dict[str, Any], +) -> CheckResult: + """Bounds-check every series returned by one parity sanity query. + + Args: + session: aiohttp client session. + prometheus_url: Prometheus API base URL. + entry: One PARITY_VALUE_SANITY entry. + + Returns: + A CheckResult that fails if any series is out of bounds, naming each + offending series. + """ + name = entry["name"] + lo = entry["lo"] + hi = entry["hi"] + exclusive_lo = entry.get("exclusive_lo", False) + check_name = f"parity.value_sanity.{name}" + + try: + async with session.get( + f"{prometheus_url}/api/v1/query", params={"query": entry["query"]} + ) as resp: + data = await resp.json() + results = data.get("data", {}).get("result", []) + + if not results: + return CheckResult( + name=check_name, + category="parity", + passed=False, + message=f"{name}: no data returned from Prometheus", + ) + + values: list[float] = [] + offenders: list[str] = [] + for series in results: + value = float(series["value"][1]) + values.append(value) + if not _value_in_bounds(value, lo, hi, exclusive_lo): + offenders.append(f"{_series_label(series)} value {value}") + + bound_desc = _bounds_description(lo, hi, exclusive_lo) + return CheckResult( + name=check_name, + category="parity", + passed=not offenders, + message=( + f"{name}: all {len(values)} series within bounds ({bound_desc})" + if not offenders + else f"{name}: {len(offenders)} of {len(values)} series out of " + f"bounds (expected {bound_desc}): " + "; ".join(offenders) + ), + details={ + "values": values, + "series_count": len(values), + "out_of_bounds": offenders, + "lo": lo, + "hi": hi, + }, + ) + except Exception as exc: + return CheckResult( + name=check_name, + category="parity", + passed=False, + message=f"{name}: sanity check failed ({exc})", + ) + + async def validate_parity_value_sanity( session: aiohttp.ClientSession, prometheus_url: str, @@ -1322,8 +1638,11 @@ async def validate_parity_value_sanity( ) -> None: """Validate that external-parity metric values fall within sane bounds. - For each entry in PARITY_VALUE_SANITY, queries the current value from - Prometheus and checks it against the specified [lo, hi] range. + For each entry in PARITY_VALUE_SANITY, queries Prometheus and checks + *every* returned series against the specified [lo, hi] range. These + queries are bare selectors with no aggregation, so a multi-node harness + cluster returns one series per service_instance_id; checking only the + first would let an out-of-range node pass silently. Args: session: aiohttp client session. @@ -1333,73 +1652,7 @@ async def validate_parity_value_sanity( logger.info("--- External Parity: Value Sanity Checks ---") for entry in PARITY_VALUE_SANITY: - name = entry["name"] - query = entry["query"] - lo = entry["lo"] - hi = entry["hi"] - exclusive_lo = entry.get("exclusive_lo", False) - check_name = f"parity.value_sanity.{name}" - - try: - params = {"query": query} - async with session.get( - f"{prometheus_url}/api/v1/query", params=params - ) as resp: - data = await resp.json() - results = data.get("data", {}).get("result", []) - - if not results: - report.add( - CheckResult( - name=check_name, - category="parity", - passed=False, - message=f"{name}: no data returned from Prometheus", - ) - ) - continue - - # Use the first result's value. - value = float(results[0]["value"][1]) - - # Check bounds. - in_range = True - if exclusive_lo: - in_range = in_range and (value > lo) - else: - in_range = in_range and (value >= lo) - if hi is not None: - in_range = in_range and (value <= hi) - - # Build human-readable bound description. - lo_op = ">" if exclusive_lo else ">=" - bound_desc = f"{lo_op} {lo}" - if hi is not None: - bound_desc += f" and <= {hi}" - - report.add( - CheckResult( - name=check_name, - category="parity", - passed=in_range, - message=( - f"{name}: value {value} is within bounds ({bound_desc})" - if in_range - else f"{name}: value {value} out of bounds " - f"(expected {bound_desc})" - ), - details={"value": value, "lo": lo, "hi": hi}, - ) - ) - except Exception as exc: - report.add( - CheckResult( - name=check_name, - category="parity", - passed=False, - message=f"{name}: sanity check failed ({exc})", - ) - ) + report.add(await _check_parity_value(session, prometheus_url, entry)) # --------------------------------------------------------------------------- diff --git a/docker/telemetry/workload/workload-profiles.json b/docker/telemetry/workload/workload-profiles.json index 84abcf3ed0..958f3524fa 100644 --- a/docker/telemetry/workload/workload-profiles.json +++ b/docker/telemetry/workload/workload-profiles.json @@ -1,7 +1,7 @@ { "profiles": { "full-validation": { - "description": "Full 18-dashboard coverage with burst/idle/plateau patterns", + "description": "Full coverage of all 15 provisioned dashboards (14 assert metric data; log-derived-insights is Loki-backed and only checked for provisioning) with burst/idle/plateau patterns across 7 phases", "phases": [ { "name": "warmup", diff --git a/docker/telemetry/workload/workload_orchestrator.py b/docker/telemetry/workload/workload_orchestrator.py index d9697d4943..5978c28e20 100755 --- a/docker/telemetry/workload/workload_orchestrator.py +++ b/docker/telemetry/workload/workload_orchestrator.py @@ -53,6 +53,28 @@ logger = logging.getLogger("workload_orchestrator") SCRIPT_DIR = Path(__file__).parent.resolve() PROFILES_FILE = SCRIPT_DIR / "workload-profiles.json" +# Wall-clock allowance for a generator on top of its phase's configured +# duration. It has to cover the work the generators do outside their timed +# loop: tx_submitter.py creates and funds 8 accounts (~25 WebSocket round +# trips) and then waits a fixed 10s for those funding transactions to +# validate, and both generators drain in-flight requests while shutting down. +# A generator that outruns this is killed and the phase records the timeout as +# an error, so one wedged process can no longer stall the whole profile. +SUBPROCESS_GRACE_SEC = 90.0 + +# How long to keep reading a killed process's output before giving up on it. +SUBPROCESS_DRAIN_TIMEOUT_SEC = 10.0 + +# Read size for the pipe readers. Only bounds one read() call, not the total. +PIPE_READ_CHUNK_BYTES = 65536 + +# Error-rate ceilings for the exit gate. The TX ceiling is higher because +# short-lived CI test environments lack pre-funded accounts, causing expected +# failures for complex transactions (AMMCreate, EscrowFinish, etc.) that +# require specific ledger state. +RPC_ERROR_RATE_LIMIT_PCT = 50.0 +TX_ERROR_RATE_LIMIT_PCT = 95.0 + # --------------------------------------------------------------------------- # Data classes @@ -146,15 +168,45 @@ def load_profile(profile_name: str) -> dict[str, Any]: # --------------------------------------------------------------------------- -async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]: - """Run a subprocess and capture its stdout and stderr. +async def _accumulate(stream: asyncio.StreamReader, chunks: list[bytes]) -> None: + """Read a subprocess pipe to EOF, appending as it goes. + + Appending to a caller-owned list, rather than returning at EOF, means + everything read so far survives even if this task never reaches EOF. Args: - cmd: Command and arguments. - label: Human-readable label for logging. + stream: Pipe to read. + chunks: List the caller reads once the process has exited. + """ + while True: + chunk = await stream.read(PIPE_READ_CHUNK_BYTES) + if not chunk: + return + chunks.append(chunk) + + +async def run_subprocess( + cmd: list[str], label: str, timeout: float +) -> tuple[int, str, str]: + """Run a subprocess to completion, or kill it once ``timeout`` expires. + + A generator that wedges used to block its phase — and so the rest of the + profile — until something outside the orchestrator killed the whole run, + destroying the report with it. Bounding the wait lets the orchestrator kill + the process, keep the output it had already produced, and report the phase + as failed. + + Both pipes are drained by separate tasks for the process's whole life, so a + chatty generator can never fill a pipe buffer and stall waiting to write. + + Args: + cmd: Command and arguments. + label: Human-readable label for logging. + timeout: Wall-clock limit in seconds. Returns: - Tuple of (return_code, stdout_text, stderr_text). + Tuple of (return_code, stdout_text, stderr_text). On timeout the return + code is non-zero and the timeout is appended to the stderr text. """ logger.debug("Starting %s: %s", label, " ".join(cmd)) proc = await asyncio.create_subprocess_exec( @@ -162,15 +214,49 @@ async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]: stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) - stdout, stderr = await proc.communicate() - if proc.returncode != 0: + + out_chunks: list[bytes] = [] + err_chunks: list[bytes] = [] + readers = [ + asyncio.create_task(_accumulate(proc.stdout, out_chunks)), + asyncio.create_task(_accumulate(proc.stderr, err_chunks)), + ] + + timed_out = False + try: + # asyncio.wait_for raises the builtin TimeoutError on Python 3.11+. + await asyncio.wait_for(proc.wait(), timeout=timeout) + except TimeoutError: + timed_out = True + logger.error("%s exceeded its %.0fs budget — killing it", label, timeout) + proc.kill() + try: + await asyncio.wait_for(proc.wait(), timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC) + except TimeoutError: + logger.error("%s did not exit after being killed", label) + + # The pipes reach EOF once the process is gone, which ends both readers. + _, pending = await asyncio.wait(readers, timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC) + for task in pending: + logger.error("%s output pipe stayed open — captured output truncated", label) + task.cancel() + + stderr_text = b"".join(err_chunks).decode(errors="replace") + if timed_out: + # Appended, not prepended: callers keep only the tail of stderr. + stderr_text += f"\ntimed out after {timeout:.0f}s and was killed" + + # A process whose exit was never collected reports no code; call it SIGKILL + # so the status is still non-zero and the phase records an error. + returncode = proc.returncode if proc.returncode is not None else -9 + if returncode != 0: logger.warning( "%s exited with code %d: %s", label, - proc.returncode, - stderr.decode().strip()[-500:], + returncode, + stderr_text.strip()[-500:], ) - return proc.returncode, stdout.decode(), stderr.decode() + return returncode, b"".join(out_chunks).decode(errors="replace"), stderr_text # --------------------------------------------------------------------------- @@ -259,6 +345,49 @@ def _build_tx_cmd( return cmd +def _launch_phase_tasks( + phase: dict[str, Any], + endpoints: list[str], + report_dir: Path, + prefix: str, +) -> list[tuple[str, Path, asyncio.Task]]: + """Start the generators this phase configures. + + Each generator is given the phase duration plus SUBPROCESS_GRACE_SEC, so a + wedged one is killed instead of stalling the phase. + + Args: + phase: Phase dict from the profile. + endpoints: List of WebSocket endpoint URLs. + report_dir: Directory for per-phase JSON reports. + prefix: Report filename prefix for this phase. + + Returns: + List of (label, report_path, task) for every generator started; empty + when the phase configures no workload. + """ + name = phase["name"] + duration = phase["duration_sec"] + timeout = duration + SUBPROCESS_GRACE_SEC + tasks: list[tuple[str, Path, asyncio.Task]] = [] + + rpc_cfg = phase.get("rpc") + if rpc_cfg: + rpc_out = report_dir / f"{prefix}-rpc.json" + cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out) + task = asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]", timeout)) + tasks.append(("rpc", rpc_out, task)) + + tx_cfg = phase.get("tx") + if tx_cfg: + tx_out = report_dir / f"{prefix}-tx.json" + cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out) + task = asyncio.create_task(run_subprocess(cmd, f"TX [{name}]", timeout)) + tasks.append(("tx", tx_out, task)) + + return tasks + + async def run_phase( phase: dict[str, Any], endpoints: list[str], @@ -292,24 +421,8 @@ async def run_phase( phase.get("description", ""), ) - tasks: list[tuple[str, Path, asyncio.Task]] = [] t0 = time.monotonic() - - rpc_cfg = phase.get("rpc") - if rpc_cfg: - rpc_out = report_dir / f"{prefix}-rpc.json" - cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out) - tasks.append( - ("rpc", rpc_out, asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]"))) - ) - - tx_cfg = phase.get("tx") - if tx_cfg: - tx_out = report_dir / f"{prefix}-tx.json" - cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out) - tasks.append( - ("tx", tx_out, asyncio.create_task(run_subprocess(cmd, f"TX [{name}]"))) - ) + tasks = _launch_phase_tasks(phase, endpoints, report_dir, prefix) if not tasks: logger.warning( @@ -406,6 +519,58 @@ async def run_profile( return report +# --------------------------------------------------------------------------- +# Exit gate +# --------------------------------------------------------------------------- + + +def evaluate_exit_gate(report: dict[str, Any]) -> list[str]: + """Decide whether a finished run should fail, and say why. + + Three independent conditions fail a run: + * a phase recorded an error — a generator exited non-zero, was killed on + timeout, or wrote a report that could not be parsed, + * the RPC error rate exceeded RPC_ERROR_RATE_LIMIT_PCT, + * the TX error rate exceeded TX_ERROR_RATE_LIMIT_PCT. + + The phase errors have to be judged separately from the two rates. A + generator that crashes writes no report, so its totals stay 0, both rates + short-circuit to 0, and a rate-only gate passes a run in which no traffic + was generated at all. + + Args: + report: Combined report produced by run_profile. + + Returns: + One human-readable reason per failure; empty when the run passed. + """ + reasons: list[str] = [] + + for phase in report.get("phases", []): + for error in phase.get("errors", []): + reasons.append(f"phase '{phase.get('name', '?')}': {error}") + + totals = report.get("totals", {}) + rpc_sent = totals.get("rpc_sent", 0) + tx_submitted = totals.get("tx_submitted", 0) + rpc_err_rate = totals.get("rpc_errors", 0) / rpc_sent * 100 if rpc_sent > 0 else 0.0 + tx_err_rate = ( + totals.get("tx_errors", 0) / tx_submitted * 100 if tx_submitted > 0 else 0.0 + ) + + if rpc_err_rate > RPC_ERROR_RATE_LIMIT_PCT: + reasons.append( + f"RPC error rate {rpc_err_rate:.1f}% exceeds " + f"{RPC_ERROR_RATE_LIMIT_PCT}% of {rpc_sent} requests" + ) + if tx_err_rate > TX_ERROR_RATE_LIMIT_PCT: + reasons.append( + f"TX error rate {tx_err_rate:.1f}% exceeds " + f"{TX_ERROR_RATE_LIMIT_PCT}% of {tx_submitted} submissions" + ) + return reasons + + # --------------------------------------------------------------------------- # CLI entry point # --------------------------------------------------------------------------- @@ -482,23 +647,12 @@ def main() -> None: json.dump(report, f, indent=2) logger.info("Combined report written to %s", args.report) - # Exit with error if either generator had high error rates. - totals = report["totals"] - rpc_err_rate = ( - totals["rpc_errors"] / totals["rpc_sent"] * 100 if totals["rpc_sent"] > 0 else 0 - ) - tx_err_rate = ( - totals["tx_errors"] / totals["tx_submitted"] * 100 - if totals["tx_submitted"] > 0 - else 0 - ) - # TX threshold is higher because short-lived CI test environments lack - # pre-funded accounts, causing expected failures for complex transactions - # (AMMCreate, EscrowFinish, etc.) that require specific ledger state. - if rpc_err_rate > 50 or tx_err_rate > 95: - logger.error( - "High error rates: RPC=%.1f%%, TX=%.1f%%", rpc_err_rate, tx_err_rate - ) + # Fail on any phase error as well as on high error rates. + failures = evaluate_exit_gate(report) + if failures: + logger.error("Workload failed %d gate condition(s):", len(failures)) + for reason in failures: + logger.error(" %s", reason) sys.exit(1) diff --git a/docker/telemetry/workload/xrpld-validator.cfg.template b/docker/telemetry/workload/xrpld-validator.cfg.template index 725faf550c..083c2eaf25 100644 --- a/docker/telemetry/workload/xrpld-validator.cfg.template +++ b/docker/telemetry/workload/xrpld-validator.cfg.template @@ -1,6 +1,10 @@ # xrpld validator node configuration template for workload harness. # -# Placeholders (replaced by docker-compose entrypoint): +# Not consumed by anything today: run-full-validation.sh writes each node's +# cfg inline. Kept as the reference layout for a validator run as a container, +# whose entrypoint would substitute the placeholders below. +# +# Placeholders: # {{NODE_INDEX}} — Node number (1-based) # {{RPC_PORT}} — HTTP RPC port # {{WS_PORT}} — WebSocket port @@ -18,16 +22,19 @@ port_rpc port_ws port_peer +# RPC and WebSocket stay on loopback, matching the cfg that +# run-full-validation.sh generates and the rest of the repo's node configs. +# Only the peer port below needs to listen on all interfaces. [port_rpc] port = {{RPC_PORT}} -ip = 0.0.0.0 -admin = 0.0.0.0 +ip = 127.0.0.1 +admin = 127.0.0.1 protocol = http [port_ws] port = {{WS_PORT}} -ip = 0.0.0.0 -admin = 0.0.0.0 +ip = 127.0.0.1 +admin = 127.0.0.1 protocol = ws [port_peer] diff --git a/docker/telemetry/xrpld-telemetry-mainnet.cfg b/docker/telemetry/xrpld-telemetry-mainnet.cfg index 9bd8779483..720c9a1b6b 100644 --- a/docker/telemetry/xrpld-telemetry-mainnet.cfg +++ b/docker/telemetry/xrpld-telemetry-mainnet.cfg @@ -9,13 +9,17 @@ # 1. Start the observability stack: # docker compose -f docker/telemetry/docker-compose.yml up -d # 2. Run xrpld: -# ./xrpld --conf docker/telemetry/xrpld-telemetry.cfg +# ./xrpld --conf docker/telemetry/xrpld-telemetry-mainnet.cfg # 3. Wait for sync (server_state=full), then exercise workflows: -# curl -s http://localhost:5005 -d '{"method":"server_info"}' +# curl -s http://localhost:5015 -d '{"method":"server_info"}' # 4. View traces in Grafana Explore -> Tempo: http://localhost:3000 # --- Server ports ----------------------------------------------------------- +# Ports are offset by +10 from the devnet config (xrpld-telemetry.cfg) so both +# nodes can run at the same time. They are host processes sharing one network +# namespace, so identical ports would leave the second node unable to bind. + [server] port_rpc_admin_local port_ws_admin_local @@ -23,24 +27,30 @@ port_ws_public port_peer [port_rpc_admin_local] -port = 5005 +port = 5015 ip = 127.0.0.1 admin = 127.0.0.1 protocol = http [port_ws_admin_local] -port = 6006 +port = 6016 ip = 127.0.0.1 admin = 127.0.0.1 protocol = ws +# Bound to loopback: this port has no `admin` key, so every caller resolves to +# Role::GUEST. Every workflow documented for this config is driven over the two +# admin ports above, which also serve WebSocket, and this node follows Mainnet, so +# there is no reason to accept off-host clients. [port_ws_public] -port = 6005 -ip = 0.0.0.0 +port = 6015 +ip = 127.0.0.1 protocol = ws +# Stays on all interfaces: this is the peer-protocol listener, and binding it to +# loopback would stop inbound overlay connections. [port_peer] -port = 51235 +port = 51245 ip = 0.0.0.0 protocol = peer @@ -73,21 +83,29 @@ validators-mainnet.txt [path_search_max] 10 -# --- Signing (allows sign/sign_for RPC for test tx submission) -------------- +# --- Signing ---------------------------------------------------------------- -[signing_support] -true +# [signing_support] is deliberately omitted (it defaults to false). It is only +# consulted for non-admin callers, and every signing path in this repo already +# runs as admin over the loopback admin ports above, so enabling it would add +# nothing except exposing sign/sign_for/channel_authorize to guests on a node +# that follows Mainnet. Upstream also deprecates these commands. # --- Database --------------------------------------------------------------- +# Paths carry the network name so this node never shares a store with the devnet +# config. Both are relative to the working directory (the repo root, per the +# usage note above), so an unqualified `data/` would have the two nodes opening +# the same NuDB and the same SQLite ledger databases — including the case where +# they run one after the other rather than concurrently. [node_db] type=NuDB -path=docker/telemetry/data/nudb +path=docker/telemetry/data/mainnet/nudb online_delete=2000 advisory_delete=0 [database_path] -docker/telemetry/data +docker/telemetry/data/mainnet [ledger_history] 1000 @@ -134,7 +152,6 @@ enabled=1 service_instance_id=xrpld-mainnet endpoint=http://localhost:4318/v1/traces metrics_endpoint=http://localhost:4318/v1/metrics -exporter=otlp_http # Mainnet has high span throughput across peer/ledger/consensus. Head # sampling is fixed at 1.0 (sample everything) and not configurable; # reduce Tempo/collector load with collector-side tail sampling. diff --git a/docker/telemetry/xrpld-telemetry.cfg b/docker/telemetry/xrpld-telemetry.cfg index bd7454e598..b362465b7a 100644 --- a/docker/telemetry/xrpld-telemetry.cfg +++ b/docker/telemetry/xrpld-telemetry.cfg @@ -125,7 +125,6 @@ enabled=1 service_instance_id=xrpld-devnet endpoint=http://localhost:4318/v1/traces metrics_endpoint=http://localhost:4318/v1/metrics -exporter=otlp_http batch_size=512 batch_delay_ms=5000 max_queue_size=2048 diff --git a/docs/build/telemetry.md b/docs/build/telemetry.md index eca4e9f110..8735a652ca 100644 --- a/docs/build/telemetry.md +++ b/docs/build/telemetry.md @@ -28,13 +28,19 @@ When enabled, it instruments RPC requests with trace spans that are exported via OTLP/HTTP to an OpenTelemetry Collector, which forwards them to a tracing backend such as Grafana Tempo. -Telemetry is **off by default** at both compile time and runtime: +Telemetry is gated twice — once at compile time and once at runtime: -- **Compile time**: The Conan option `telemetry` and CMake option `telemetry` must be set to `True`/`ON`. - When disabled, all `SpanGuard` calls compile to inline no-ops (defined in `SpanGuard.h`) +- **Compile time**: The Conan option `telemetry` and CMake option `telemetry` decide + whether the OTel SDK is linked in and `XRPL_ENABLE_TELEMETRY` is defined. + When off, all `SpanGuard` calls compile to inline no-ops (defined in `SpanGuard.h`) with zero overhead — no OTel SDK dependency required. -- **Runtime**: The `[telemetry]` config section must set `enabled=1`. - When disabled at runtime, a no-op implementation is used. + The option is currently `True`/`ON` on the telemetry branches so that CI builds and + exercises the instrumented code; **`False`/`OFF` is the intended default once this + feature is merged.** Pass the value you want explicitly rather than relying on the + default. +- **Runtime**: Telemetry is **off by default** — the `[telemetry]` config section must + set `enabled=1`. When disabled at runtime, a no-op implementation is used even in a + build that has the SDK compiled in. ## Building with Telemetry @@ -102,12 +108,22 @@ cmake --build . --parallel $(nproc) ## Building without telemetry -Omit the `-o telemetry=True` option (or pass `-o telemetry=False`). +Pass `-o telemetry=False` to `conan install`, and `-Dtelemetry=OFF` to CMake if you +configure without the Conan-generated toolchain. Do not just omit the option — it then +resolves to whatever the current default is, and that default is `True` on the +telemetry branches. + The `opentelemetry-cpp` dependency will not be downloaded, the `XRPL_ENABLE_TELEMETRY` preprocessor define will not be set, and all tracing macros will compile to no-ops. The resulting binary is identical to one built before telemetry support was added. +> **`-DXRPL_ENABLE_TELEMETRY=OFF` disables nothing.** `XRPL_ENABLE_TELEMETRY` is not a +> CMake option — it is only a compile definition added when `telemetry` is on. Passing it +> on the command line leaves telemetry compiled in; CMake merely lists it at the end of +> configuration under `Manually-specified variables were not used by the project`. +> Use `-Dtelemetry=OFF`. + ## Troubleshooting ### Conan lockfile error diff --git a/docs/telemetry-glossary.md b/docs/telemetry-glossary.md index b5a5a64559..a0ffe14863 100644 --- a/docs/telemetry-glossary.md +++ b/docs/telemetry-glossary.md @@ -9,12 +9,12 @@ documentation. > **Related docs**: > [docs/telemetry-runbook.md](./telemetry-runbook.md) (operator runbook). - + ## Contents @@ -665,7 +665,7 @@ How many jobs of a given type are queued or executing at the instant the queue i ### Ledger acquire (inbound fetch) -Acquiring a ledger means requesting it and its contents from peers when the node lacks it. Acquire outcomes split into complete and failed; a rising failed rate means the node cannot fetch needed ledgers from its peers. +Acquiring a ledger means requesting it and its contents from peers when the node lacks it. Acquire outcomes split three ways: complete, failed (the acquisition ended on its own without the ledger, having run out of retries or hit unusable data), and aborted (it was abandoned before finishing, either swept away as stale or discarded wholesale at shutdown). A rising failed rate means the node cannot fetch needed ledgers from its peers. **Scope:** per node — measured on and specific to this individual server. @@ -1021,9 +1021,9 @@ A cluster is a set of servers run by the same operator that trust each other, ex **Scope:** cluster-wide — shared across a co-operated cluster of nodes run by one operator. -**What is observable:** cluster overhead is **not** measurable today. Cluster messages are counted under `unknown` rather than `overhead_cluster`, so the `overhead_cluster_*` series read zero on a clustered node — treat them as "no data", not "no cluster traffic". The churn guidance above cannot yet be acted on. +**What is observable:** cluster overhead is **not** measurable today, and this is a gap in the instrumentation rather than a display problem. The cluster message type is not in the overlay's message-to-category lookup table and none of the fallback branches match it, so every cluster message falls through to the `unknown` category (`src/xrpld/overlay/detail/TrafficCount.cpp`); no code path ever reports the cluster category, even though the `overhead_cluster` name is defined. Consequences: the `overhead_cluster_*` series read zero on a clustered node — treat them as "no data", not "no cluster traffic" — the churn guidance above cannot yet be acted on, and `unknown_*` is a weaker anomaly signal on a clustered node because it mixes genuinely unrecognized wire types with routine cluster traffic. If this is ever instrumented, volume moves out of `unknown_bytes_in`, so any alert threshold set against that series will need re-baselining. -**See also:** [Cluster on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/clustering) · [Data collection reference §6.0](../OpenTelemetryPlan/09-data-collection-reference.md#60-mtcluster-is-counted-as-unknown-not-implemented) +**See also:** [Cluster on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/clustering) @@ -1101,9 +1101,7 @@ Each peer connection is probed on a timer: the node sends a ping carrying a rand **Scope:** per node — measured on and specific to this individual server. -**What is observable:** only the p90 of the smoothed per-peer latency (`peer_quality{metric="peer_latency_p90_ms"}`) — there is no distribution, ping timeouts and wrong-cookie pongs have no counter, and ping bytes are not separable from status-change bytes because both share the `overhead` traffic category. - -**See also:** [Data collection reference §6.3](../OpenTelemetryPlan/09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) +**What is observable:** only the p90 of the smoothed per-peer latency (`peer_quality{metric="peer_latency_p90_ms"}`), and three things are not instrumented. First, the per-peer round-trip is an 8-sample moving average and is exported as that single p90 gauge, with no histogram — a bimodal peer set (a few very slow peers behind many fast ones) reads as one middling number. Second, neither failure mode is counted: a ping timeout only logs before dropping the peer, and a pong bearing the wrong cookie is discarded silently (`src/xrpld/overlay/detail/PeerImp.cpp`), so keepalive-driven drops cannot be separated from any other disconnect cause. Third, ping bytes are not separable from status-change bytes, because both message types share the `overhead` traffic category — so `overhead_*` cannot be read as keepalive volume. Peer discovery traffic is separable (it lands in `overhead_overlay_*`) but has no counters of its own for endpoints received, handed out or malformed. @@ -1181,9 +1179,7 @@ Squelching is a relay-control mechanism: a node tells peers to stop sending it a **Scope:** per node — measured on and specific to this individual server. -**What is observable:** read ignored directives on `squelch_ignored_messages_in/out` only. The paired `squelch_ignored_bytes_*` series are always zero because the ignored-squelch callback records no size, so bandwidth wasted by peers ignoring squelch cannot be quantified — and `squelch_ignored` is therefore not comparable on bytes against `squelch_suppressed`, which does record real sizes. - -**See also:** [Data collection reference §6.1](../OpenTelemetryPlan/09-data-collection-reference.md#61-squelch_ignored-byte-counts-not-implemented) +**What is observable:** read ignored directives on `squelch_ignored_messages_in` only. The size is not instrumented: both call sites that report an ignored squelch pass a hardcoded byte count of zero (`src/xrpld/overlay/detail/OverlayImpl.cpp`), so `squelch_ignored_bytes_in` is always zero and the bandwidth wasted by peers ignoring squelch cannot be quantified, nor can a bytes-per-message ratio be built from this category. `squelch_suppressed` does record the real wire size, so the two squelch categories are not comparable on bytes — only on message counts. The outbound side of this category is never reported at all, so `squelch_ignored_bytes_out` and `squelch_ignored_messages_out` are also permanently zero; that is expected, since "ignoring a squelch" is something a remote peer does to us and is therefore only ever observed inbound. diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index e8eea7bfe0..c70ae15c69 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -81,12 +81,16 @@ endpoint=http://localhost:4318/v1/traces ### 3. Build with telemetry support +Follow [BUILD.md](../BUILD.md), adding `-o telemetry=True` so Conan pulls `opentelemetry-cpp`. From a build directory (`.build/`): + ```bash -conan install . --build=missing -o telemetry=True -cmake --preset default -Dtelemetry=ON -cmake --build --preset default +conan install .. --output-folder . --build missing -o telemetry=True --settings build_type=Release +cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Dxrpld=ON -Dtelemetry=ON .. +cmake --build . --target xrpld ``` +Conan also writes a `conan-release` CMake preset, so `cmake --preset conan-release -Dtelemetry=ON` works instead of the explicit toolchain line. There is no preset named `default`. + ### 4. Run against a live network Two ready-made configs connect a tracking node (no validator credentials) to a @@ -112,7 +116,7 @@ Metrics begin flowing as soon as the node connects to peers (`server_state` (`server_state` = `full`). Check progress with: ```bash -curl -s http://localhost:5005 -d '{"method":"server_info"}' | +curl -s http://localhost:5015 -d '{"method":"server_info"}' | jq '.result.info | {server_state, peers, complete_ledgers}' ``` @@ -122,25 +126,32 @@ curl -s http://localhost:5005 -d '{"method":"server_info"}' | ## Configuration Reference -| Option | Default | Description | -| -------------------------- | --------------------------------- | --------------------------------------------------------- | -| `enabled` | `0` | Master switch for telemetry | -| `endpoint` | `http://localhost:4318/v1/traces` | OTLP/HTTP endpoint | -| `service_name` | `xrpld` | OpenTelemetry service name resource attribute | -| `service_instance_id` | node public key | OpenTelemetry service instance ID resource attribute | -| `trace_rpc` | `1` | Enable RPC request tracing | -| `trace_transactions` | `1` | Enable transaction tracing | -| `trace_consensus` | `1` | Enable consensus tracing | -| `trace_peer` | `1` | Enable peer message tracing (high volume) | -| `trace_ledger` | `1` | Enable ledger tracing | -| `consensus_trace_strategy` | `deterministic` | Consensus trace ID strategy (`deterministic` or `random`) | -| `batch_size` | `512` | Max spans per batch export | -| `batch_delay_ms` | `5000` | Delay between batch exports | -| `max_queue_size` | `2048` | Max spans queued before dropping | -| `use_tls` | `0` | Use TLS for exporter connection | -| `tls_ca_cert` | (empty) | Path to CA certificate bundle | -| `tls_client_cert` | (empty) | Client cert (PEM) for mutual TLS; empty = one-way TLS | -| `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert` | +| Option | Default | Description | +| -------------------------- | --------------------------------- | ------------------------------------------------------------ | +| `enabled` | `0` | Master switch for telemetry | +| `endpoint` | `http://localhost:4318/v1/traces` | OTLP/HTTP endpoint | +| `service_name` | `xrpld` | OpenTelemetry service name resource attribute | +| `service_instance_id` | node public key | OpenTelemetry service instance ID resource attribute | +| `trace_rpc` | `1` | Enable RPC request tracing | +| `trace_transactions` | `1` | Enable transaction tracing | +| `trace_consensus` | `1` | Enable consensus tracing | +| `trace_peer` | `1` | Enable peer message tracing (high volume) | +| `trace_ledger` | `1` | Enable ledger tracing | +| `consensus_trace_strategy` | `deterministic` | Consensus trace ID strategy (`deterministic` or `attribute`) | +| `batch_size` | `512` | Max spans per batch export | +| `batch_delay_ms` | `5000` | Delay between batch exports | +| `max_queue_size` | `2048` | Max spans queued before dropping | +| `use_tls` | `0` | Use TLS for exporter connection | +| `tls_ca_cert` | (empty) | Path to CA certificate bundle | +| `tls_client_cert` | (empty) | Client cert (PEM) for mutual TLS; empty = one-way TLS | +| `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert` | + +> **`consensus_trace_strategy` is not validated.** The parser copies the raw +> string through (`TelemetryConfig.cpp:155-156`) and the only equality test in +> the code is `strategy == "attribute"` (`RCLConsensus.cpp:1296`). Any other +> value — including a typo such as `determinstic` — silently selects the +> deterministic branch. There is no warning in the log. The two accepted values +> are documented at `include/xrpl/telemetry/Telemetry.h:287-292`. ## Exporting to Grafana Cloud @@ -217,9 +228,8 @@ to a datasource of the matching type — auto-selecting it when only one exists (the usual case: one Mimir, one Tempo). This is what makes the same files work unchanged on both the local stack and Cloud. -> Dashboards are parameterized by `grafana/parameterize-datasources.py`. If you -> add a dashboard exported with hardcoded UIDs, re-run that script (idempotent) -> before committing so it stays portable. +> If you add a dashboard exported with hardcoded datasource UIDs, replace them +> with `${DS_PROMETHEUS}` / `${DS_TEMPO}` before committing. To import: @@ -242,17 +252,28 @@ All spans instrumented in xrpld, grouped by subsystem: | -------------------- | ----------------- | ----------------------------------------------------------- | ----------------------------------------------------- | | `rpc.http_request` | ServerHandler.cpp | `request_payload_size` | Top-level HTTP RPC request | | `rpc.ws_upgrade` | ServerHandler.cpp | — | WebSocket upgrade handshake | -| `rpc.ws_message` | ServerHandler.cpp | `command` | WebSocket RPC message | +| `rpc.ws_message` | ServerHandler.cpp | `command`, `rpc_status` | WebSocket RPC message | | `rpc.process` | ServerHandler.cpp | `is_batch`, `batch_size` | RPC processing (child of rpc.http_request/ws_message) | | `rpc.command.` | RPCHandler.cpp | `command`, `version`, `rpc_role`, `rpc_status`, `load_type` | Per-command span (e.g., `rpc.command.server_info`) | +On `rpc.ws_message`, `rpc_status` is set **on four of the five error paths** +(resource threshold exceeded, bad API version / missing command, caught +exception, and an error in the command result — `ServerHandler.cpp:489`, `:522`, +`:571`, `:608`). The exception is the **invalid-JSON / oversized-request** path, +which opens its own `rpc.ws_message` span and calls only `setError()`, writing no +`rpc_status` at all (`ServerHandler.cpp:392-395`) — those rejections are visible +solely through `status_code="ERROR"`. The success path calls `setOk()` and writes +no `rpc_status` either, so there is never an `rpc_status="success"` series for +this span: count successes as total minus error, or filter on `status_code`. +`rpc.command.*` is unaffected — it sets `rpc_status` on both outcomes. + ### Transaction Spans | Span Name | Source File | Attributes | Description | | --------------- | --------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `tx.process` | NetworkOPs.cpp | `tx_hash`, `local`, `path`, `tx_type`, `fee`, `sequence`, `ter_result`, `applied`, `current_ledger_seq` | Transaction submission and processing | | `tx.receive` | PeerImp.cpp | `peer_id`, `tx_hash`, `tx_type`, `peer_version`, `suppressed`, `tx_status`, `current_ledger_seq` | Transaction received from peer relay | -| `tx.apply` | BuildLedger.cpp | `ledger_seq`, `tx_count`, `tx_failed` | Transaction set applied per ledger | +| `tx.apply` | BuildLedger.cpp | `tx_count`, `tx_failed` | Transaction set applied per ledger | | `tx.preflight` | applySteps.cpp | `stage`, `tx_type`, `ter_result` | Stateless checks stage | | `tx.preclaim` | applySteps.cpp | `stage`, `tx_type`, `ter_result`, `current_ledger_seq`, `current_ledger_hash` | Ledger-aware checks stage | | `tx.transactor` | Transactor.cpp | `stage`, `tx_type`, `ter_result`, `applied`, `current_ledger_seq`, `current_ledger_hash` | Apply stage (transactor runs) | @@ -270,6 +291,11 @@ txID-keyed spans can be joined to the ledger trace it targeted `tx.transactor`) also carry `current_ledger_hash` (the current ledger's parent hash); `tx.preflight` is stateless and omits both. +`tx.apply` carries **no** `ledger_seq` of its own — the sequence is set on its +parent `ledger.build` +([BuildLedger.cpp:90](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L90)), so +read it from the parent rather than filtering `tx.apply` on it. + ### Transaction Queue Spans | Span Name | Source File | Attributes | Description | @@ -283,37 +309,56 @@ hash); `tx.preflight` is stateless and omits both. ### PathFinding Spans -| Span Name | Source File | Attributes | Description | -| --------------------- | --------------------------------- | -------------------------------------------------- | ------------------------------------------------------- | -| `pathfind.request` | PathFind.cpp / RipplePathFind.cpp | `pathfind_source_account`, `pathfind_dest_account` | Path-find RPC entry (accounts hashed; set when present) | -| `pathfind.compute` | PathRequest.cpp | `pathfind_fast`, `pathfind_dest_currency` | Path computation for one request (`doUpdate`) | -| `pathfind.discover` | PathRequest.cpp | `pathfind_search_level`, `pathfind_num_paths` | Graph exploration (one per RPC call in `findPaths`) | -| `pathfind.update_all` | PathRequestManager.cpp | `pathfind_ledger_index`, `pathfind_num_requests` | Async recomputation of active requests on ledger close | +| Span Name | Source File | Attributes | Description | +| --------------------- | --------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------- | +| `pathfind.request` | PathFind.cpp / RipplePathFind.cpp | `pathfind_source_account`, `pathfind_dest_account` | Path-find RPC entry (accounts hashed; set when present) | +| `pathfind.compute` | PathRequest.cpp | `pathfind_fast`, `pathfind_dest_currency` | Path computation for one request (`doUpdate`) | +| `pathfind.discover` | PathRequest.cpp | `pathfind_search_level`, `pathfind_num_paths`, `pathfind_num_source_assets` | Graph exploration (one per RPC call in `findPaths`) | +| `pathfind.update_all` | PathRequestManager.cpp | `pathfind_ledger_index`, `pathfind_num_requests` | Async recomputation of active requests on ledger close | ### Consensus Spans -| Span Name | Source File | Attributes | Description | -| ------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| `consensus.round` | RCLConsensus.cpp | `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id` | Root span for a consensus round (deterministic or random trace ID) | -| `consensus.phase.open` | Consensus.h | -- | Open phase duration (child of round) | -| `consensus.proposal.send` | RCLConsensus.cpp | `consensus_round`, `is_bow_out` | Consensus proposal broadcast | -| `consensus.ledger_close` | RCLConsensus.cpp | `ledger_seq`, `consensus_mode` | Ledger close event | -| `consensus.establish` | Consensus.h | `converge_percent`, `establish_count`, `proposers` | Establish phase duration (child of round) | -| `consensus.update_positions` | Consensus.h | `converge_percent`, `proposers`, `disputes_count` | Position update and dispute resolution (see Events below) | -| `consensus.check` | Consensus.h | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, `consensus_stalled`, `establish_count`, `consensus_result` | Consensus threshold check | -| `consensus.accept` | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` | Ledger accepted by consensus | -| `consensus.accept.apply` | RCLConsensus.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `tx_count` | Ledger application with close time details (see Events below) | -| `consensus.validation.send` | RCLConsensus.cpp | `ledger_seq`, `proposing`, `ledger_hash`, `full_validation`, `validation_sign_time` | Validation sent after accept (follows-from link) | -| `consensus.mode_change` | RCLConsensus.cpp | `mode_old`, `mode_new` | Consensus mode transition | -| `consensus.proposal.receive` | PeerImp.cpp | `proposal_trusted`, `consensus_round` | Proposal received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | -| `consensus.validation.receive` | PeerImp.cpp | `validation_trusted`, `ledger_seq` | Validation received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | +| Span Name | Source File | Attributes | Description | +| ------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `consensus.round` | RCLConsensus.cpp | `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id` | Root span for a consensus round (deterministic or random trace ID) | +| `consensus.phase.open` | Consensus.h | `open_duration_ms`, `peer_positions_at_close` (both only if the span is still live at `closeLedger()`) | Open phase duration (child of round) | +| `consensus.proposal.send` | RCLConsensus.cpp | `consensus_round`, `is_bow_out` | Consensus proposal broadcast | +| `consensus.ledger_close` | RCLConsensus.cpp | `ledger_seq`, `consensus_mode` | Ledger close event | +| `consensus.establish` | Consensus.h | `converge_percent`, `establish_count`, `proposers` | Establish phase duration (child of round) | +| `consensus.update_positions` | Consensus.h | `converge_percent`, `proposers`, `disputes_count`, `avalanche_threshold` (only when peer positions exist), `have_close_time_consensus`, `close_time_threshold` | Position update and dispute resolution (see Events below) | +| `consensus.check` | Consensus.h | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, `consensus_stalled`, `establish_count`, `consensus_result` | Consensus threshold check | +| `consensus.accept` | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` | Ledger accepted by consensus | +| `consensus.accept.apply` | RCLConsensus.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `tx_count`, `disputes_resolved_count` | Ledger application with close time details (see Events below) | +| `consensus.validation.send` | RCLConsensus.cpp | `ledger_seq`, `proposing`, `ledger_hash`, `full_validation`, `validation_sign_time` | Validation sent after accept (follows-from link) | +| `consensus.mode_change` | RCLConsensus.cpp | `mode_old`, `mode_new` | Consensus mode transition | +| `consensus.proposal.receive` | PeerImp.cpp | `proposal_trusted`, `consensus_round`, `prev_ledger_prefix`, `position_hash_prefix` | Proposal received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | +| `consensus.validation.receive` | PeerImp.cpp | `validation_trusted`, `ledger_seq` (only when the validation carries `sfLedgerSequence`), `full_validation`, `validation_sign_time` | Validation received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | #### Consensus Span Events -| Parent Span | Event Name | Event Attributes | Description | -| ---------------------------- | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------- | -| `consensus.update_positions` | `dispute.resolve` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | Emitted per dispute when votes are tallied | -| `consensus.accept.apply` | `tx.included` | `tx_id` | Emitted per transaction included in the accepted ledger | +| Parent Span | Event Name | Event Attributes | Description | +| ---------------------------- | ------------------ | ----------------------------------------------------------- | -------------------------------------------------------- | +| `consensus.update_positions` | `dispute.resolve` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | Emitted per dispute when votes are tallied | +| `consensus.accept.apply` | `tx.included` | `tx_id` | Emitted per transaction included in the accepted ledger | +| `consensus.round` | `phase.open` | -- | Round entered the open phase (also re-fired on recovery) | +| `consensus.round` | `phase.recovery` | -- | Round started with `StartRoundReason::Recovered` | +| `consensus.round` | `phase.establish` | -- | Round entered the establish phase on close | +| `consensus.round` | `phase.accepted` | -- | Round reached the accepted phase | +| `consensus.round` | `outcome.yes` | -- | Round settled with consensus reached | +| `consensus.round` | `outcome.moved_on` | -- | Round abandoned; the network moved on without us | +| `consensus.round` | `outcome.expired` | -- | Round expired without settling | + +The nine events above are the complete set. The seven on `consensus.round` +carry **no event attributes** — they are timestamps marking phase entry and the +terminal outcome, so a round's whole life reads off one span's event list. +Phase entry additionally rewrites the round's span-level `consensus_phase` +attribute, which is why `phase.recovery` is the one phase event that leaves +`consensus_phase` unchanged (it fires with an empty label). Evidence: +[RCLConsensus.cpp:1344](../src/xrpld/app/consensus/RCLConsensus.cpp#L1344), +[1386](../src/xrpld/app/consensus/RCLConsensus.cpp#L1386), +[1400](../src/xrpld/app/consensus/RCLConsensus.cpp#L1400); outcomes are chosen +from `result_->state` at +[Consensus.h:1517-1525](../include/xrpl/consensus/Consensus.h#L1517). #### Close Time Queries (Tempo TraceQL) @@ -341,18 +386,64 @@ Span attributes are filtered with `span.` inside `{}`. Combine conditions ### Ledger Spans -| Span Name | Source File | Attributes | Description | -| ----------------- | -------------------- | ------------------------------------- | ----------------------------- | -| `ledger.build` | BuildLedger.cpp:31 | `ledger_seq`, `tx_count`, `tx_failed` | Ledger build during consensus | -| `ledger.validate` | LedgerMaster.cpp:915 | `ledger_seq`, `validations` | Ledger promoted to validated | -| `ledger.store` | LedgerMaster.cpp:409 | `ledger_seq` | Ledger stored in history | +| Span Name | Source File | Attributes | Description | +| ----------------- | ----------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| `ledger.build` | BuildLedger.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms` | Ledger build during consensus | +| `ledger.validate` | LedgerMaster.cpp | `ledger_seq`, `validations` | Ledger promoted to validated | +| `ledger.store` | LedgerMaster.cpp | `ledger_seq` | Ledger stored in history | +| `ledger.acquire` | InboundLedger.cpp | `ledger_seq`, `acquire_reason`, `timeouts`, `peer_count`, `outcome` | Fetch a missing ledger from peers (parent varies — see [known issues](#where-telemetry-parenting-differs-from-protocol-flow)) | + +`ledger.acquire` sets only `ledger_seq` and `acquire_reason` when the span opens +in `init()`. `outcome` has three values, written on two different paths: + +| `outcome` | Written where | Meaning | +| ---------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `complete` | `done()` | The ledger was fetched. | +| `failed` | `done()` | The acquisition ended on its own without the ledger. Usually it gave up after `timeouts_ > kLedgerTimeoutRetriesMax` (= 6), but `trigger()` also fails immediately on an unusable state or transaction map, so a `failed` span can carry `timeouts=0`. Carries span status `Error`. | +| `aborted` | destructor | The acquisition was abandoned before finishing — the sweep evicted it a minute after anything last asked for it, or `ledgers_` was cleared wholesale by shutdown or by `clearFailures()`. Status is left `Unset`, because the shutdown case is benign. | + +`peer_count` is written only on the `done()` path, so it is absent on `aborted` +spans: reading it would go through `Overlay`, which a destructor running at +teardown cannot depend on still existing. `timeouts` is written on both paths. + +A missing `outcome` has two causes, and neither is a lost span. The common one is +that `init()` satisfied the ledger straight from the local store, so the acquire +never went to the network. The other is a hard failure inside `tryDB()`: a stored +header that cannot be this ledger, or a zero account hash, sets `failed_` and +`init()` returns without ever calling `done()`, so no outcome is written. The +destructor does not fill the gap either — its `if (!isDone())` guard is already +false once `failed_` is set, because `isDone()` is `complete_ || failed_`. Such a +span carries `ledger_seq` and `acquire_reason` only. Since `aborted` exists, a +missing `outcome` is no longer how an abandoned acquisition presents. + +When reading acquire **duration**, exclude or split out `outcome="aborted"`. +Those spans stay open from `init()` until the object is destroyed, so they measure +how long the acquisition stayed outstanding rather than fetch latency, and will +skew a percentile that mixes them with `complete`. Only on the sweep path is that +duration bounded below by the one-minute threshold. The shutdown and +`clearFailures()` paths abort at whatever age the acquisition happened to have, so +an `aborted` span can also be arbitrarily short. + +`ledger.build` does **not** carry `tx_count` / `tx_failed`. Those two live on its +child `tx.apply` span, which is where the set is actually applied +([BuildLedger.cpp:191](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L191)) — +join on the trace, not on one span. + +> **Gap: no `ledger.*` span carries a ledger hash.** The attribute constant +> `ledger_span::attr::ledgerHash` is declared +> ([LedgerSpanNames.h:41](../src/xrpld/app/ledger/detail/LedgerSpanNames.h#L41)) +> but is never set by any call site, so `ledger.build` / `ledger.store` / +> `ledger.validate` / `ledger.acquire` are identifiable by `ledger_seq` only. A +> query filtering on `span.ledger_hash` over a `ledger.*` span returns nothing. +> `ledger_hash` **is** set on `consensus.validation.send` and on the peer spans, +> so use those when a hash is required. ### Peer Spans -| Span Name | Source File | Attributes | Description | -| ------------------------- | ---------------- | ------------------------------- | ----------------------------- | -| `peer.proposal.receive` | PeerImp.cpp:1667 | `peer_id`, `proposal_trusted` | Proposal received from peer | -| `peer.validation.receive` | PeerImp.cpp:2264 | `peer_id`, `validation_trusted` | Validation received from peer | +| Span Name | Source File | Attributes | Description | +| ------------------------- | ----------- | ------------------------------- | ----------------------------- | +| `peer.proposal.receive` | PeerImp.cpp | `peer_id`, `proposal_trusted` | Proposal received from peer | +| `peer.validation.receive` | PeerImp.cpp | `peer_id`, `validation_trusted` | Validation received from peer | Both peer receive spans are `kConsumer` inbound entry points started as fresh trace roots. They never inherit an ambient span left active on the peer thread, @@ -694,11 +785,11 @@ in `Establish` across many heartbeats until the outcome is decided. > 1. **Avalanche rounds inside one Establish phase** — each `timerEntry` runs > `phaseEstablish` again (`establishCounter_++`) and raises the inclusion > threshold **50% → 65% → 70% → 95%** as the round ages -> ([ConsensusParms.h:145](../src/xrpld/consensus/ConsensusParms.h#L145)). +> ([ConsensusParms.h:145](../include/xrpl/consensus/ConsensusParms.h#L145)). > `checkConsensus` returning `No` keeps the node in `Establish` and loops; a > round cannot even `Expire` before a minimum of > `avalancheCutoffs.size() × avMinRounds = 4 × 2 = 8` passes -> ([Consensus.h:1938](../src/xrpld/consensus/Consensus.h#L1938)). +> ([Consensus.h:1937](../include/xrpl/consensus/Consensus.h#L1937)). > 2. **Retry across consensus rounds** — a round can end `MovedOn` / `Expired`, > meaning the network settled a _different_ ledger. The node still builds a > ledger, but the **next** round's `checkLedger` detects the wrong prior, @@ -760,44 +851,44 @@ Consensus loops and branches (evidence): - **`consensus.establish` is the parent of `update_positions` and `check`**: `phaseEstablish` creates the establish span (`startEstablishTracing`), and both child spans parent to its captured context - ([Consensus.h:2100](../src/xrpld/consensus/Consensus.h#L2100), - [1629](../src/xrpld/consensus/Consensus.h#L1629), - [1838](../src/xrpld/consensus/Consensus.h#L1838)). + ([Consensus.h:2099](../include/xrpl/consensus/Consensus.h#L2099), + [1628](../include/xrpl/consensus/Consensus.h#L1628), + [1837](../include/xrpl/consensus/Consensus.h#L1837)). - **Avalanche-convergence loop (rounds within one ledger)**: repeated `heartbeat → timerEntry → phaseEstablish` bumps `establishCounter_` and raises the inclusion threshold each pass; `checkConsensus` = `No` stays in `Establish` ([NetworkOPs.cpp:1214](../src/xrpld/app/misc/NetworkOPs.cpp#L1214); - [Consensus.h:1468](../src/xrpld/consensus/Consensus.h#L1468); - thresholds [ConsensusParms.h:145](../src/xrpld/consensus/ConsensusParms.h#L145)). + [Consensus.h:1467](../include/xrpl/consensus/Consensus.h#L1467); + thresholds [ConsensusParms.h:145](../include/xrpl/consensus/ConsensusParms.h#L145)). - **Retry-across-rounds loop (many rounds per settled ledger)**: `MovedOn` / `Expired` accepts a non-preferred ledger; the next round's `checkLedger` finds the wrong prior and recovers before re-deliberating - ([Consensus.h:1194](../src/xrpld/consensus/Consensus.h#L1194)); round-to-round + ([Consensus.h:1193](../include/xrpl/consensus/Consensus.h#L1193)); round-to-round via `endConsensus → beginConsensus` ([NetworkOPs.cpp:2315](../src/xrpld/app/misc/NetworkOPs.cpp#L2315)). - **Two extra establish loop-backs before accept**: `shouldPause` (laggard backpressure) and `!haveCloseTimeConsensus_` (TX consensus but not close-time) each `return` and re-loop, distinct from `checkConsensus == No` - ([Consensus.h:1497](../src/xrpld/consensus/Consensus.h#L1497), - [1500](../src/xrpld/consensus/Consensus.h#L1500)); close time can + ([Consensus.h:1496](../include/xrpl/consensus/Consensus.h#L1496), + [1499](../include/xrpl/consensus/Consensus.h#L1499)); close time can "agree to disagree" at prior close + 1s ([docs/consensus.md:163](consensus.md)). - **acquireTxSet / gotTxSet loop**: a disagreeing peer position triggers an async `acquireTxSet`; the later `gotTxSet` regenerates disputes and can extend the - establish phase ([Consensus.h:932](../src/xrpld/consensus/Consensus.h#L932)). + establish phase ([Consensus.h:931](../include/xrpl/consensus/Consensus.h#L931)). - **Bow-out / mode change**: `handleWrongLedger → leaveConsensus` sends a bow-out proposal and demotes Proposing → Observing for the rest of the round - ([Consensus.h:1977](../src/xrpld/consensus/Consensus.h#L1977)); `startRound` + ([Consensus.h:1976](../include/xrpl/consensus/Consensus.h#L1976)); `startRound` begins in Proposing **or** Observing ([docs/consensus.md:176](consensus.md)). - **Buffered Open-phase inputs**: `peerProposal` / `gotTxSet` arriving during Open are stored, then seeded as disputes at `closeLedger` (`createDisputes`); `playbackProposals` replays them at `startRound` / `handleWrongLedger` ([docs/consensus.md:244](consensus.md); - [Consensus.h:817](../src/xrpld/consensus/Consensus.h#L817)). + [Consensus.h:816](../include/xrpl/consensus/Consensus.h#L816)). - **Outcome fork** after `checkConsensus`: `No` (loop) / `Yes` (onAccept) / - `MovedOn` / `Expired` ([Consensus.h:1516](../src/xrpld/consensus/Consensus.h#L1516)). + `MovedOn` / `Expired` ([Consensus.h:1515](../include/xrpl/consensus/Consensus.h#L1515)). - **Expired guard**: a round cannot leave on `Expired` before `avalancheCutoffs.size() × avMinRounds` (= 8) passes — below that, `Expired` - loops like `No` ([Consensus.h:1938](../src/xrpld/consensus/Consensus.h#L1938)). + loops like `No` ([Consensus.h:1937](../include/xrpl/consensus/Consensus.h#L1937)). - The **deterministic-vs-random trace-strategy** branch at round start ([RCLConsensus.cpp:1291](../src/xrpld/app/consensus/RCLConsensus.cpp#L1291)) sets only the trace ID — it has **zero protocol effect**. @@ -920,10 +1011,11 @@ flowchart TB UALL -.->|dead / aborted| DEAD ``` -**Ledger acquire** — a **separate trace root** (not part of the close flow) that -fetches a missing or correct-prior ledger from peers, retries per peer/timer, and -finishes with a reason-dependent store; `checkAccept` + `tryAdvance` run on **any** -completed acquire: +**Ledger acquire** — a flow **outside the close flow** that fetches a missing or +correct-prior ledger from peers, retries per peer/timer, and finishes with a +reason-dependent store; `checkAccept` + `tryAdvance` run on **any** completed +acquire. `ledger.acquire` is usually a trace root, but not reliably so — see the +[parenting known issues](#where-telemetry-parenting-differs-from-protocol-flow): ```mermaid flowchart TB @@ -965,13 +1057,41 @@ Side-flow evidence: [181](../src/xrpld/rpc/detail/PathRequestManager.cpp#L181)). - **Acquire outcome fork**: `timeouts_ > kLedgerTimeoutRetriesMax` (= 6) sets `failed_` → terminal `logFailure`, no store/checkAccept - ([InboundLedger.cpp:387](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L387)). + ([InboundLedger.cpp:402](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L402)). + A third path never reaches `done()` at all: the destructor marks any acquisition + that is still neither `complete_` nor `failed_` as `outcome=aborted` + ([InboundLedgers.cpp:393](../src/xrpld/app/ledger/detail/InboundLedgers.cpp#L393) + sweep eviction; [InboundLedger.cpp:224](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L224) + abort branch). Give-up fires at roughly **18s**, not 21s: `init()` enters the + retry loop through `queueJob()` with no preceding `setTimer()`, so the first + `invokeOnTimer()` runs immediately with `progress_` still `false` and takes + `timeouts_` to 1 at t≈0. The test needs `timeouts_ > 6` — the seventh invocation + — and only six 3s intervals separate the seventh from the first, so 6 x 3s = 18s. + A live `aborted` rate does **not** by itself mean acquisitions are stalling. + Three unrelated paths produce it: + - **Sweep eviction** — the only cause that implies staleness, and it fires a + minute after anything last _asked for_ this ledger, not a minute after the + last byte arrived. + - **Shutdown** — `InboundLedgers::stop()` clears `ledgers_` wholesale, so every + clean stop aborts every acquisition still in flight. + - **`clearFailures()`** — also clears `ledgers_`, and is reachable at runtime + from the `fetch_info` admin RPC (`clear: true` → + `NetworkOPsImp::clearLedgerFetch()`), so an operator can produce aborts on a + perfectly healthy node. + + The 18s-vs-60s gap does not settle it either: while the acquisition lane sits at + its job limit the timer body never runs, so `timeouts_` cannot advance and the + give-up path is disarmed exactly when aborts are likeliest — see + [The deferral/timeout pair](#the-deferraltimeout-pair). Rule out shutdown and + `clearFailures()` first, then read a sustained `aborted` rate against + `acquire_sweep_evictions`. + - **done() reason branch (store side only)**: `HISTORY` → `onLedgerFetched`, **no** `storeLedger`; else → `storeLedger`. But `checkAccept` + `tryAdvance` run for **any** `complete_ && !failed_` acquire regardless of reason - ([InboundLedger.cpp:495](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L495) - store switch; [507](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L507) - reason-independent checkAccept/tryAdvance). + ([InboundLedger.cpp:537](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L537) + store switch; [552](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L552) + reason-independent checkAccept/tryAdvance on the `AcqDone` job). - **tryAdvance multi-ledger loop**: `doAdvance` runs `do { … } while (advanceWork_)`, publishing a range of ledgers and recursively triggering further HISTORY acquire ([LedgerMaster.cpp:1905](../src/xrpld/app/ledger/detail/LedgerMaster.cpp#L1905)). @@ -982,22 +1102,88 @@ The graph above is protocol control flow. The OpenTelemetry span **parent links* are built differently and, in several places, do **not** represent a real call edge. Read a trace with these in mind: -| Telemetry does this | Real protocol flow | -| ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `tx.process` is a `hashSpan` root from `txID` — an independent trace root ([TxTracing.h:63](../src/xrpld/telemetry/TxTracing.h#L63)). | The real edge is the synchronous `doSubmit → processTransaction` call; it is **not** a child of `rpc.command.submit`. | -| `tx.preflight` / `tx.preclaim` / `tx.transactor` share one `txID`-derived trace ID. | That shared ID is a correlation trick, not a call edge. The real order is the composed `apply()` at [apply.cpp:118](../src/libxrpl/tx/apply.cpp#L118). They are **not** children of `tx.process` or `tx.apply`. | -| `consensus.round` uses a deterministic trace ID from the previous ledger hash. | This makes **all validators share one trace ID** (a cross-node shared root), not a per-node parent. The real round-to-round edge is `endConsensus → beginConsensus`. | -| `consensus.accept` (main thread) and `consensus.accept.apply` (JtAccept worker) are wired via a captured context. | The real edge is the queued `JtAccept` job, a thread hand-off ([RCLConsensus.cpp:483](../src/xrpld/app/consensus/RCLConsensus.cpp#L483)). | -| `pathfind.update_all` parents nothing from the original `pathfind.request`. | The causal link is the ledger-close job on `JtUpdatePf`, not span nesting. | -| `ledger.acquire` and its downstream `ledger.store` / `ledger.validate`. | Reached via the `AcqDone` job, not parent inheritance; `ledger.acquire` is its own root. | -| `peer.*.receive` (fresh `kConsumer` root) and `consensus.*.receive` on the same message. | Two **sequential stages of one synchronous handler**, not parent/child; on a duplicate/untrusted drop the `consensus.*.receive` is never created. | -| Receive spans adopt the sender's `trace_id` + `span_id` as a genuine cross-node parent. | Deliberate: the receive span becomes a child of a **different node's** span (a cross-node context marker, not an in-process edge). `tx.receive` is asymmetric — it borrows only the sender's `span_id` and re-derives its own `trace_id` from `txID`. | +| Telemetry does this | Real protocol flow | +| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `tx.process` is a `hashSpan` root from `txID` — an independent trace root ([TxTracing.h:63](../src/xrpld/telemetry/TxTracing.h#L63)). | The real edge is the synchronous `doSubmit → processTransaction` call; it is **not** a child of `rpc.command.submit`. | +| `tx.preflight` / `tx.preclaim` / `tx.transactor` share one `txID`-derived trace ID. | That shared ID is a correlation trick, not a call edge. The real order is the composed `apply()` at [apply.cpp:118](../src/libxrpl/tx/apply.cpp#L118). They are **not** children of `tx.process` or `tx.apply`. Because nothing else nests under it either, `tx.apply` is **always a leaf** — the stage spans for the transactions it applied sit in the txID-keyed trace, not beneath it. | +| `consensus.round` uses a deterministic trace ID from the previous ledger hash. | This makes **all validators share one trace ID** (a cross-node shared root), not a per-node parent. The real round-to-round edge is `endConsensus → beginConsensus`. | +| `consensus.accept` (main thread) and `consensus.accept.apply` (JtAccept worker) are wired via a captured context. | The real edge is the queued `JtAccept` job, a thread hand-off ([RCLConsensus.cpp:483](../src/xrpld/app/consensus/RCLConsensus.cpp#L483)). | +| `pathfind.update_all` parents nothing from the original `pathfind.request`. | The causal link is the ledger-close job on `JtUpdatePf`, not span nesting. | +| `ledger.acquire` and its downstream `ledger.store` / `ledger.validate`. | Reached via the `AcqDone` job, not parent inheritance. All three are non-scoped `SpanGuard::span` spans, so none of them parents the others; each takes whatever ambient span its own caller happens to have active. See the `ledger.*` known issue below. | +| `peer.*.receive` (fresh `kConsumer` root) and `consensus.*.receive` on the same message. | Two **sequential stages of one synchronous handler**, not parent/child; on a duplicate/untrusted drop the `consensus.*.receive` is never created. | +| Receive spans adopt the sender's `trace_id` + `span_id` as a genuine cross-node parent. | Deliberate: the receive span becomes a child of a **different node's** span (a cross-node context marker, not an in-process edge). `tx.receive` is asymmetric — it borrows only the sender's `span_id` and re-derives its own `trace_id` from `txID`. | > **Known telemetry artifacts** (from live audits, memory `otel-span-hierarchy-audit`): > an RPC entry span's scope can leak across a reused coroutine worker, and the -> `hashSpan` roots (`tx.*`) — along with plain roots like `ledger.acquire` — can -> surface in Tempo as dangling "root span not yet received". These are -> exporter/parenting artifacts, not real control-flow parents. +> `hashSpan` roots (`tx.*`) — along with `ledger.acquire` / `ledger.store` / +> `ledger.validate` whenever they do come out parentless — can surface in Tempo +> as dangling "root span not yet received". These are exporter/parenting +> artifacts, not real control-flow parents. + +Three further divergences are **known issues in the code**, not deliberate design. +Unlike the rows above, these produce a parent that is simply wrong, and all three +are pending a code fix: + +- **`grpc.*` and `pathfind.update_all` do not open a fresh root.** All four RPC + entry points create their span with `freshRoot`, so a reused coroutine worker + cannot leak a stale ambient parent into them + ([ServerHandler.cpp:473](../src/xrpld/rpc/detail/ServerHandler.cpp#L473), + [640](../src/xrpld/rpc/detail/ServerHandler.cpp#L640)). `grpc.` + ([GRPCServer.cpp:173](../src/xrpld/app/main/GRPCServer.cpp#L173)) and + `pathfind.update_all` + ([PathRequestManager.cpp:91](../src/xrpld/rpc/detail/PathRequestManager.cpp#L91)) + use the plain constructor instead, so either can be adopted by whatever span + happened to be active on the worker that picked the job up. A gRPC call + appearing beneath an unrelated transaction's trace is this bug, not a real + call edge. +- **`ledger.acquire` / `ledger.store` / `ledger.validate` are not reliably roots + either.** All three use `SpanGuard::span` + ([InboundLedger.cpp:113](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L113), + [LedgerMaster.cpp:463](../src/xrpld/app/ledger/detail/LedgerMaster.cpp#L463), + [987](../src/xrpld/app/ledger/detail/LedgerMaster.cpp#L987)), which inherits the + ambient span ([SpanGuard.cpp:233](../src/libxrpl/telemetry/SpanGuard.cpp#L233)) + rather than `freshRoot` + ([245](../src/libxrpl/telemetry/SpanGuard.cpp#L245)) — the same defect as + `grpc.*` above. Whether they come out as roots depends purely on the caller: + - **Root, as documented.** On the `JtAdvance` / `AcqDone` job path + (`LedgerMaster::doAdvance`, `RCLConsensus::Adaptor::acquireLedger` → + [RCLConsensus.cpp:171](../src/xrpld/app/consensus/RCLConsensus.cpp#L171)) no + span is active on the worker, so nothing is inherited. `acquireSpan_` itself is + a non-scoped `SpanGuard`, so it never becomes the ambient parent of the + `ledger.store` / `ledger.validate` that follow it. + - **Mis-parented.** `InboundLedgers::acquire` is also called **synchronously from + an RPC handler** — `ledger_request` → `rpc::getOrAcquireLedger` + ([RPCLedgerHelpers.cpp:483](../src/xrpld/rpc/detail/RPCLedgerHelpers.cpp#L483)) + — which runs inside the scoped `rpc.command.` span + ([RPCHandler.cpp:168](../src/xrpld/rpc/detail/RPCHandler.cpp#L168)). There + `ledger.acquire` becomes a child of that RPC command, and when `init()` is + satisfied from the local store the `ledger.store` / `ledger.validate` it calls + ([InboundLedger.cpp:164](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L164), + [168](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L168)) land there as + siblings. A ledger acquisition nested under an `rpc.command.*` trace is this + bug, not a real call edge. + + **`ledger.build` and `tx.apply` use the same ambient-parent construct but are + safe.** `ledger.build` is a plain `ScopedSpanGuard` + ([BuildLedger.cpp:55](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L55)): its + only callers are `RCLConsensus::doAccept` + ([RCLConsensus.cpp:935-937](../src/xrpld/app/consensus/RCLConsensus.cpp#L935)) + on the `JtAccept` worker and the replay path + ([LedgerDeltaAcquire.cpp:208](../src/xrpld/app/ledger/detail/LedgerDeltaAcquire.cpp#L208)), + and every consensus accept span is a non-scoped `SpanGuard` + ([RCLConsensus.cpp:598-599](../src/xrpld/app/consensus/RCLConsensus.cpp#L598)), + so no ambient span exists to be inherited there. `tx.apply` + ([BuildLedger.cpp:123](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L123)) is + reached only synchronously from `buildLedgerImpl` while `ledger.build`'s scope is + live, so its ambient parent is always `ledger.build` — which is exactly the + intended edge. + +- **`consensus.round` is not always a root.** The `consensus_trace_strategy=attribute` + path has two creation branches; the fallback branch — taken on the first traced + round of a run, and whenever consensus tracing is off — sets no parent at all + ([RCLConsensus.cpp:1310](../src/xrpld/app/consensus/RCLConsensus.cpp#L1310)), + so that round inherits the ambient context instead of starting a trace. Rounds + under the default `deterministic` strategy are unaffected. --- @@ -1122,10 +1308,20 @@ sum by (stage) (rate(span_calls_total{span_name=~"tx.preflight|tx.preclaim|tx.tr > a rising `tx.transactor` failure rate points to apply-time problems. Alert per > stage rather than on a single aggregate so the failing stage is obvious. -> **Sampling caveat**: these stage metrics are span-derived and inherit the -> **tracer head-sampling** ratio (`sampling_ratio`). At `sampling_ratio < 1.0` -> they undercount proportionally — treat them as relative trends, not absolute -> transaction counts. Native StatsD metrics are unsampled. +> **Sampling caveat**: these stage metrics are span-derived, but head sampling +> is **fixed at 100% and is not configurable** — the ratio is a compile-time +> constant ([Telemetry.h:234](../include/xrpl/telemetry/Telemetry.h#L234) +> `static constexpr double samplingRatio = 1.0;`) and there is no +> `sampling_ratio` config key to set +> ([TelemetryConfig.cpp:139](../src/libxrpl/telemetry/TelemetryConfig.cpp#L139) +> — "nothing to parse"). So locally these counts are **exact**, not a sample. +> Volume reduction is a collector-side **tail** sampling decision instead, and +> the only policy shipped is a single 0.5% probabilistic one that lives **only** +> in `otel-collector-config.grafanacloud.yaml` — the base +> `otel-collector-config.yaml` has no tail sampling at all, so a stock local +> stack retains every trace. Where that Cloud policy is in force it applies to +> the trace-storage branch only; spanmetrics run on a separate branch and still +> see 100% of spans, so the derived RED metrics stay exact either way. ### Transaction Queue Health @@ -1450,7 +1646,7 @@ Note that `job_count` is exported as `jobq_job_count`: the JobQueue is constructed with `collectorManager_->group("jobq")` (Application.cpp:386), `GroupImp::makeName()` joins prefix and name with a `.` (Groups.cpp:42), and `OTelCollectorImp::formatName()` then turns the `.` into `_` and lowercases the -whole string (OTelCollector.cpp:860). The same mechanism produces the +whole string (OTelCollector.cpp:855-874). The same mechanism produces the `jobq_{jobtype}_*` names above and the pre-existing `jobq_{jobtype}_milliseconds` timing family. @@ -1597,9 +1793,9 @@ ledger acquisition deferring". Use `acquire_ledger_deferrals` and These five come from the `PerfLog` job hooks, not from beast::insight, so they are exported by the `MetricsRegistry` meter. `job_queued_us` and `job_running_us` have explicit microsecond bucket views registered -(`addMicrosecondHistogramView()`, MetricsRegistry.cpp:253-254) spanning 100 µs to -60 s; without those the SDK default buckets stop at 10 ms and every quantile -saturates. +(`addMicrosecondHistogramView()` calls at MetricsRegistry.cpp:310-311; the helper +itself is at `:197`) spanning 100 µs to 60 s; without those the SDK default +buckets stop at 10 ms and every quantile saturates. | Prometheus Metric | Kind | Labels | Description | | -------------------- | --------- | --------------------- | ------------------------------------ | @@ -1776,9 +1972,7 @@ line added to `addMicrosecondHistogramView()` in `MetricsRegistry.cpp` -- the only case that still touches a central file. There is no way to read a metric's current value back from application code -- OTel's API is write-only by design; keep your own state if your logic needs to both record and read a running value -(see the Doxygen header in `MetricMacros.h` and "Use Case 4" in -`tasks/metric-macro-plan.md` for the full explanation and the `prometheus-cpp` -contrast rationale). +(see the Doxygen header in `MetricMacros.h` for the full explanation). ## Deployment Tiers @@ -1881,11 +2075,16 @@ Fifteen dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/` Fourteen are Prometheus-backed; `log-derived-insights` is the only Loki/LogQL board and is documented last, together with the LogQL-specific traps it exposed. -> Nine dashboards have a reference section below. `fee-market`, `job-queue`, -> `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality`, and -> `validator-health` are provisioned but not yet documented here — their panel -> descriptions carry the same six-heading reference format, so open the panel -> info icon in Grafana until a section is written. +> **Nine of the fifteen have a reference section.** Eight are in this chapter +> (`rpc-performance`, `transaction-overview`, `consensus-health`, +> `ledger-operations`, `peer-network`, `node-health`, `network-traffic`, +> `rpc-pathfinding`); the ninth, `log-derived-insights`, is documented under +> [Log-Trace Correlation](#log-derived-insights-log-derived-insights). The +> remaining **six** — `fee-market`, `job-queue`, `ledger-data-sync`, +> `overlay-traffic-detail`, `peer-quality`, and `validator-health` — are +> provisioned but not yet documented here. Their panel descriptions carry the same +> six-heading reference format, so open the panel info icon in Grafana until a +> section is written. ### RPC Performance (`rpc-performance`) @@ -1895,7 +2094,7 @@ board and is documented last, together with the LogQL-specific traps it exposed. | RPC Latency p95 by Command | timeseries | `histogram_quantile(0.95, sum by (le, command) (rate(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `command` | | RPC Error Rate | bargauge | Error spans / total spans × 100, grouped by `command` | `command`, `status_code` | | RPC Latency Heatmap | heatmap | `sum(increase(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)` | `le` (bucket boundaries) | -| Overall RPC Throughput | timeseries | `rpc.request` + `rpc.process` rate | — | +| Overall RPC Throughput | timeseries | `rpc.http_request` + `rpc.process` rate | — | | RPC Success vs Error | timeseries | by `status_code` (UNSET vs ERROR) | `status_code` | | Top Commands by Volume | bargauge | `topk(10, ...)` by `command` | `command` | | WebSocket Message Rate | stat | `rpc.ws_message` rate | — | @@ -1961,19 +2160,19 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — | | Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — | | Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | -| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | -| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | +| Operating Mode Transitions | timeseries | `increase(state_accounting_*_transitions[$__rate_interval])` | — | +| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_milliseconds_bucket)` | — | | Job Queue Depth | timeseries | `jobq_job_count` | — | | Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — | -| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — | -| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | -| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` | +| Ledger History Mismatches | stat | `rate(ledger_history_mismatch_total[5m])` | — | +| Key Jobs Execution Time | timeseries | `histogram_quantile($quantile, sum by (le) (rate(job_running_us_bucket{job_type="acceptLedger"}[$__rate_interval])))` (+ 10 more key jobs) | `job_type` | +| Key Jobs Dequeue Wait Time | timeseries | `histogram_quantile($quantile, sum by (le) (rate(job_queued_us_bucket{job_type="acceptLedger"}[$__rate_interval])))` (+ 10 more) | `job_type` | | FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — | | FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — | | Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — | | State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — | -| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"", quantile="$quantile"}` | `quantile` | -| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"_q", quantile="$quantile"}` | `quantile` | +| All Jobs Execution Time (Detail) | timeseries | `histogram_quantile($quantile, sum by (le, job_type) (rate(job_running_us_bucket[$__rate_interval])))` | `job_type` | +| All Jobs Dequeue Wait (Detail) | timeseries | `histogram_quantile($quantile, sum by (le, job_type) (rate(job_queued_us_bucket[$__rate_interval])))` | `job_type` | | Server State | stat | `server_info{metric="server_state"}` | `metric` | | Uptime | stat | `server_info{metric="uptime"}` | `metric` | | Peer Count | stat | `server_info{metric="peers"}` | `metric` | @@ -1984,6 +2183,24 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | Database Sizes | timeseries | `db_metrics{metric=~"db_kb_.*"}` | `metric` | | Historical Fetch Rate | stat | `db_metrics{metric="historical_perminute"}` | `metric` | +> **The four job panels read the `MetricsRegistry` histograms fed by the PerfLog +> job hooks, not the `jobq_*` ones.** +> `$quantile` is a dashboard template variable holding a fraction (`0.95`), fed +> straight into `histogram_quantile()`. There is **no `quantile` label** on any +> xrpld series — that was a StatsD-era summary convention, and a selector like +> `{quantile="$quantile"}` matches nothing and reports no error. The job queue +> exposes two parallel families: `job_running_us` / `job_queued_us` +> (`MetricsRegistry` instruments, labelled by `job_type` and `handler`, +> microseconds — what these panels use; +> [MetricsRegistry.cpp:94-95](../src/xrpld/telemetry/MetricsRegistry.cpp#L94), +> [363-366](../src/xrpld/telemetry/MetricsRegistry.cpp#L363), recorded from the +> `PerfLog` job hooks at +> [PerfLogImp.cpp:432](../src/xrpld/perflog/detail/PerfLogImp.cpp#L432)) and +> `jobq_[_q]_milliseconds` +> (beast::insight, one instrument per job type, milliseconds — +> [JobTypeData.h:97](../include/xrpl/core/JobTypeData.h#L97)). Both are live; +> prefer the labelled `job_*_us` pair so one query covers every job type. + ### Network Traffic -- System Metrics (`network-traffic`) | Panel | Type | PromQL | Labels Used | @@ -2011,16 +2228,26 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ### RPC & Pathfinding -- System Metrics (`rpc-pathfinding`) -| Panel | Type | PromQL | Labels Used | -| ------------------------- | ---------- | ------------------------------------------------ | ----------- | -| RPC Request Rate | stat | `rate(rpc_requests[5m])` | — | -| RPC Response Time | timeseries | `histogram_quantile(0.95, rpc_time_bucket)` | — | -| RPC Response Size | timeseries | `histogram_quantile(0.95, rpc_size_bucket)` | — | -| RPC Response Time Heatmap | heatmap | `rpc_time_bucket` | — | -| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_bucket)` | — | -| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_bucket)` | — | -| Resource Warnings Rate | stat | `rate(warn_total[$__rate_interval])` | — | -| Resource Drops Rate | stat | `rate(drop_total[$__rate_interval])` | — | +| Panel | Type | PromQL | Labels Used | +| ------------------------- | ---------- | ------------------------------------------------------------- | ----------- | +| RPC Request Rate | stat | `rate(rpc_requests[5m])` | — | +| RPC Response Time | timeseries | `histogram_quantile(0.95, rpc_time_milliseconds_bucket)` | — | +| RPC Response Size | timeseries | `histogram_quantile(0.95, rpc_size_milliseconds_bucket)` | — | +| RPC Response Time Heatmap | heatmap | `rpc_time_milliseconds_bucket` | — | +| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_milliseconds_bucket)` | — | +| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_milliseconds_bucket)` | — | +| Resource Warnings Rate | stat | `rate(warn_total[$__rate_interval])` | — | +| Resource Drops Rate | stat | `rate(drop_total[$__rate_interval])` | — | + +> **The `_milliseconds` suffix comes from the exporter, not from xrpld.** These +> histograms are created with unit `"ms"` +> ([OTelCollector.cpp:615](../src/libxrpl/beast/insight/OTelCollector.cpp#L615)), +> so the Prometheus exporter appends the unit to the family name — `rpc_time` +> becomes `rpc_time_milliseconds_bucket`. Querying the bare `rpc_time_bucket`, +> `ios_latency_bucket` or `pathfind_fast_bucket` returns no data and no error. +> **Known issue**: `rpc_size` counts bytes but shares the same `"ms"` histogram +> constructor, so it is exported as `rpc_size_milliseconds_bucket` — the suffix +> is wrong, the name is nonetheless the one to query. ### Span → Metric → Dashboard Summary @@ -2056,6 +2283,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | `ledger.build` | `{span_name="ledger.build"}` | Ledger Ops (Build Rate, Duration, Heatmap) | | `ledger.validate` | `{span_name="ledger.validate"}` | Ledger Ops (Validation Rate) | | `ledger.store` | `{span_name="ledger.store"}` | Ledger Ops (Store Rate) | +| `ledger.acquire` | `{span_name="ledger.acquire"}` | -- (available but not paneled) | | `peer.proposal.receive` | `{span_name="peer.proposal.receive"}` | Peer Network (Rate, Trusted/Untrusted) | | `peer.validation.receive` | `{span_name="peer.validation.receive"}` | Peer Network (Rate, Trusted/Untrusted) | @@ -2096,7 +2324,7 @@ Alerts fire only after the condition holds for the `for` dwell time. | `NodeStateFlapping` | warning | > 3 re-entries into FULL per hour | 15m | | `NodeNotFull` | warning | `server_state` < 4 (FULL) | 15m | | `ManifestJobQueueConvoy` | warning | `jobq_manifest_waiting` > 3 | 10m | -| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 512 kB/s | 10m | +| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 512 KiB/s | 10m | | `PeerResourceDisconnects` | warning | > 5 resource-driven peer disconnects per 30m | 5m | Two expression idioms recur and are load-bearing — do not "simplify" them away: @@ -2117,6 +2345,19 @@ from the validated network chain. Likely causes: corrupted local state, a bug, or a node that fell out of sync and rebuilt incorrectly. Investigate the node's ledger acquisition logs; a healthy node never mismatches. +> **Query trap — `sum(ledger_history_mismatch_total)` double-counts.** One +> mismatch increments **two** instruments inside the same `handleMismatch()` +> call: the legacy beast::insight counter, which carries no `reason` label +> ([LedgerHistory.cpp:323](../src/xrpld/app/ledger/LedgerHistory.cpp#L323)), and +> the `MetricsRegistry` counter, which does +> ([LedgerHistory.cpp:331](../src/xrpld/app/ledger/LedgerHistory.cpp#L331)). +> Both normalise to the same Prometheus family, so an unfiltered `sum()` or +> `increase()` reports exactly **twice** the real mismatch count. Aggregate over +> the labelled series only — `sum by (reason) (...)`, or +> `sum(ledger_history_mismatch_total{reason!=""})` — and halve any historical +> figure taken from the unfiltered form. The alert rule is unaffected: it only +> tests `> 0`. This is a known issue; the duplicate producer awaits a code fix. + **LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the process is hung. This rule also fires on _NoData_ — if the series disappears the @@ -2216,7 +2457,8 @@ This is the most reliable manifest-flood signal because `jobq_manifest_waiting` is `0` at the 99.9th percentile on every node over 24h — any sustained backlog is a genuine outlier rather than normal variance. -**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 512 kB/s. Catches the +**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 512 KiB/s (524288 +B/s — the rule's literal `params: [524288]`). Catches the wire-level cause (a peer shipping oversized dumps) even when the job pool absorbs it without a visible backlog. Measured over 7 days: healthy p95 0.2-0.5 kB/s and p99 1.0-1.8 kB/s, against peaks up to 2.7 MB/s during real storms — so the @@ -2344,25 +2586,33 @@ docker compose -f docker/telemetry/docker-compose.yml exec renderer \ #### Deploying alerts to Grafana Cloud Grafana Cloud has **no provisioning filesystem**, so these `apiVersion: 1` files -cannot be loaded there. Cloud deployment goes through the REST API via -`docker/telemetry/upload_alerts_to_grafana.py`, which reads the same tracked -`rules.yaml` as the single source of truth (so local and Cloud cannot drift) and -applies the Cloud-specific transforms: the local `prometheus` datasource uid is -swapped for the Cloud one, the `folder:` _name_ becomes an existing `folderUID`, -and `interval` becomes integer seconds. +cannot be loaded there. Cloud deployment goes through the Grafana alerting **REST +API**, driven from the same tracked `rules.yaml` — it stays the single source of +truth, so local and Cloud cannot drift. -```bash -cd docker/telemetry -python3 upload_alerts_to_grafana.py --dry-run # always dry-run first -python3 upload_alerts_to_grafana.py # create rules, paused -python3 upload_alerts_to_grafana.py --verify # read back what is deployed -``` +Each rule needs three Cloud-specific transforms on the way out: + +| Field in `rules.yaml` | Cloud form | +| --------------------------------- | ------------------------ | +| local `prometheus` datasource uid | the Cloud datasource uid | +| `folder:` _name_ | an existing `folderUID` | +| `interval` (duration string) | integer seconds | + +Then, in order: + +1. **Dry-run first** — render what would be sent and review it before writing + anything to the Cloud stack. +2. **Create the rules paused**, so nothing can fire on a threshold that has not + been reviewed against this fleet. +3. **Read back** the deployed rules and verify they are what was sent. + +Land the rules with delivery disabled while no recipient has been chosen, and +activate them only once the thresholds have been checked against the target +fleet's baseline. Credentials come from `.env.grafanaserviceapi` (gitignored, a service-account token with `alert.rules:write`); the recipient address comes from `ALERT_EMAIL_TO` -in `.env.alerting`. Neither is ever written to a tracked file. Use -`--no-delivery` to land the rules before a recipient is chosen, and `--activate` -only once the thresholds have been checked against the target fleet's baseline. +in `.env.alerting`. Neither is ever written to a tracked file. > **The Cloud notification policy tree must not be pushed.** There is exactly one > policy tree per org and the PUT endpoint **replaces it wholesale**. On a shared @@ -2408,7 +2658,7 @@ curl -sG http://localhost:9090/api/v1/query \ ## Log-Trace Correlation -When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: +When xrpld is built with `telemetry=ON`, log lines emitted within an active, sampled OpenTelemetry span automatically include `trace_id` and `span_id` fields: ``` 2024-Jan-15 10:30:45.123456 UTC LedgerMaster:NFO trace_id=abc123def456789012345678abcdef01 span_id=0123456789abcdef Validated ledger 42 @@ -2425,6 +2675,8 @@ Log files are ingested by the OTel Collector's `filelog` receiver, which tails ` The receiver tails `/var/log/xrpld/*/debug.log` inside the collector container. docker-compose bind-mounts the host log root there; the source defaults to the repo-relative `docker/telemetry/data/logs`, which the telemetry configs write to (`data/logs//debug.log`) and which needs no root. To tail logs from elsewhere, set `XRPLD_LOG_DIR` before `docker compose up` (the integration test does this to point at its own workdir). The single trailing `*` matches one per-network or per-node subdirectory. +Each file is read from the beginning, because the receiver's own default (`end`) would skip anything a node wrote before the collector's first poll and would never read a log that has stopped being written to. Read offsets are held in memory by default, so a restarted collector re-reads the files it already ingested. The developer stack avoids that by layering `otel-collector-filestorage.yaml` as a second `--config`, which adds a `file_storage` extension that keeps the offsets on a named volume; a one-shot init service prepares that volume, because the collector runs as a non-root user and a fresh Docker volume is owned by root. Ephemeral stacks such as the workload validation harness create a fresh log directory per run, so they have nothing to resume from and deliberately omit the overlay. + ### LogQL Query Examples The OTel Collector emits logs to Loki with `service_name="xrpld"` (not `job="xrpld"`). @@ -2518,8 +2770,8 @@ Filters: `$service_name`, `$deployment_environment`, `$node`, #### LogQL traps this dashboard exposed -Ten mistakes that fail **silently** — each cost a debugging cycle, so check them -before adding any LogQL panel. +Eleven mistakes that fail **silently** — each cost a debugging cycle, so check +them before adding any LogQL panel. 1. **`partition` is structured metadata, not a stream label.** `{service_name="xrpld", partition="ManifestCache"}` returns **zero rows with @@ -3086,7 +3338,7 @@ Two more pairs from the same family: cumulative object-payload bytes this process has written — the same value as `node_written_bytes`, from the same accessor — so it excludes keys, padding and the log, and it resets with the process. A ratio of the two is a constant 1.0 and - measures nothing. This label value was called `nudb_bytes` before Phase 9; it + measures nothing. This label value was called `nudb_bytes` in earlier revisions; it comes from `node_store::Database` rather than the NuDB backend, so it is not part of the `nudb_*` family above and reads the same on RocksDB. - These gauges are sampled on the `MetricsRegistry` reader's 10 s cadence, while @@ -4121,17 +4373,27 @@ spans are genuinely the same ledger rather than a trace-id coincidence. ## Disabling Telemetry -Set `enabled=0` in config (runtime disable) or build without the flag: +Set `enabled=0` in the `[telemetry]` config section (runtime disable, no rebuild), or +compile telemetry out: ```bash -cmake --preset default -Dtelemetry=OFF +conan install .. --output-folder . --build missing -o telemetry=False --settings build_type=Release +cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Dtelemetry=OFF .. ``` +Pass the flag explicitly rather than omitting it — an omitted flag resolves to whatever +the build's current default is. That default is `ON` on the telemetry branches so CI +compiles the instrumented paths, and `OFF` once the feature is merged; `-Dtelemetry=OFF` +is correct either way. `-DXRPL_ENABLE_TELEMETRY=OFF` does **not** work: that name is only +a compile definition added when `telemetry` is ON, not a CMake option, so telemetry stays +compiled in and CMake only lists it under `Manually-specified variables were not used by +the project`. + When telemetry is compiled out, all trace macros expand to no-ops with zero overhead. ## Validating Telemetry Stack -After deploying telemetry, use the Phase 10 workload tools to validate the full stack end-to-end. +After deploying telemetry, use the workload tools in `docker/telemetry/workload/` to validate the full stack end-to-end. ### Quick Validation @@ -4141,16 +4403,44 @@ docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld # Check the report: cat /tmp/xrpld-validation/reports/validation-report.json | jq '.summary' + +# Tear the stack and the node processes down: +docker/telemetry/workload/run-full-validation.sh --cleanup ``` +Harness options (`run-full-validation.sh`): + +| Flag | Default | Effect | +| ------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `--xrpld PATH` | `.build/xrpld` | Binary to run. Also settable via the `XRPLD` env var. | +| `--nodes NUM` | `5` | Size of the local validator cluster. | +| `--profile NAME` | `full-validation` | Load profile from `workload-profiles.json` (`full-validation`, `quick-smoke`, `stress`). This is the **only** thing that sets load shape. | +| `--skip-loki` | off | Skip the log-trace correlation checks. CI always passes this. | +| `--skip-regression` | off | Skip timing capture and the baseline comparison. Local exploration only. | +| `--with-benchmark` | off | Also run `benchmark.sh` (telemetry-off vs telemetry-on overhead) after validation. | +| `--cleanup` | — | Tear everything down and exit. | + +`--rpc-rate`, `--rpc-duration`, `--tx-tps` and `--tx-duration` are accepted by the +parser but **never read** — they predate profiles and have no effect. Use +`--profile`, or add a profile to `workload-profiles.json`. + +Exit codes: `0` all checks and the regression gate passed; `1` a validation check +failed or the gate detected a regression; `2` infrastructure error (stack or +cluster did not come up, or timing capture failed). + ### What Gets Validated -| Category | Checks | Description | -| ---------- | -------------- | ------------------------------------------------------- | -| Spans | 16+ span types | All span names appear in Tempo with required attributes | -| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics | -| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works | -| Dashboards | 15 dashboards | All Grafana dashboards load without errors | +The counts are not hard-coded in the validator — it iterates the inventory files, +so those files are authoritative. The figures below are the inventory as it +stands today. + +| Category | Checks | Description | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Spans | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true` | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions). | +| Metrics | Every entry in every asserted category of `expected_metrics.json` — 58 metrics in 23 categories at the time of writing | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has no `metrics` key, so the validator skips it. | +| Logs | 2 checks | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`. | +| Parity | 10 checks | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds. | +| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 16 provisioned dashboards | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered. | ### Running Individual Tools @@ -4171,8 +4461,134 @@ python3 docker/telemetry/workload/validate_telemetry.py \ ### Interpreting Failures - **Span failures**: Check that the relevant trace category is enabled in `[telemetry]` config (e.g., `trace_rpc=1`). -- **Metric failures**: Verify the OTel Collector is running and Prometheus is scraping port 8889. Check `docker compose logs otel-collector`. -- **Dashboard failures**: Ensure Grafana provisioning is mounted correctly. Check `docker compose logs grafana`. +- **Metric failures**: Verify the OTel Collector is running and Prometheus is scraping port 8889. +- **Dashboard failures**: Ensure Grafana provisioning is mounted correctly. + +`run-full-validation.sh` brings the stack up with +`docker compose -f docker/telemetry/docker-compose.workload.yaml`, so a bare +`docker compose logs` from the repository root finds no project. Pass the same +compose file: + +```bash +docker compose -f docker/telemetry/docker-compose.workload.yaml logs otel-collector +docker compose -f docker/telemetry/docker-compose.workload.yaml logs grafana +docker compose -f docker/telemetry/docker-compose.workload.yaml ps +``` + +### Regression Gate and CI + +The validation checks answer "is the telemetry there?". A second, independent +gate answers "did xrpld get slower?" — it is the part of this harness that can +fail CI on a performance change, so it is worth understanding before you push. + +It runs as step 6 of `run-full-validation.sh`, after validation, and is skipped +only with `--skip-regression`: + +```mermaid +flowchart TB + classDef stage fill:#1d4ed8,stroke:#1e3a8a,color:#fff; + classDef data fill:#047857,stroke:#064e3b,color:#fff; + classDef gate fill:#b45309,stroke:#7c2d12,color:#fff; + classDef out fill:#334155,stroke:#0f172a,color:#fff; + + PROM[("Prometheus
localhost:9090")]:::data + MET["regression-metrics.json
(spans + job_queue groups)"]:::data + CAP["capture_timings.py
--window REGRESSION_WINDOW"]:::stage + TIM["reports/timings.json
(key to value + unit)"]:::data + BASE["baselines/baseline-timings.json
(committed)"]:::data + THR["regression-thresholds.json
(pct AND abs bounds)"]:::data + CMP["compare_to_baseline.py"]:::stage + PH{"baseline is a placeholder
or has no metrics?"}:::gate + PASTE["Print paste-me JSON
exit 0 — gate does NOT run"]:::out + DIFF["Diff per metric
regression = over BOTH bounds"]:::gate + REP["reports/regression-report.json
exit 1 on any regression"]:::out + + MET --> CAP + PROM --> CAP --> TIM --> CMP + BASE --> CMP + THR --> CMP + CMP --> PH + PH -->|yes| PASTE + PH -->|no| DIFF --> REP +``` + +Key properties: + +- **A metric regresses only when it exceeds BOTH the percentage and the absolute + bound.** The `AND` is deliberate: SpanMetrics latency histograms use explicit + buckets, so a quantile sitting near a low bucket boundary can jump a whole + bucket (1 ms to 5 ms) with no real change. Bounds live in + `regression-thresholds.json` — `defaults` per category and quantile, with + per-metric `overrides` (e.g. `span.consensus.ledger_close` is held to 5%). +- **A metric with no configured threshold is captured but never gates.** It is + reported with a note instead. Today only `span.*` and `job.*` keys have + thresholds; `rpc.*` is not produced and would not gate if it were (see + `docker/telemetry/workload/baselines/README.md`). +- **A metric missing from the current run is not a regression.** + `summary.missing_in_current` in `regression-report.json` is a count; the + identities are the `metrics[]` entries whose `note` is + `"not captured in current run"`. +- **`REGRESSION_WINDOW`** (env var, default `3m`) is the window handed to + Prometheus `rate()` during capture. Keep it close to the workload duration — + a longer window dilutes a short-lived regression. `BASELINE_FILE`, + `THRESHOLDS_FILE` and `METRICS_FILE` are also env-overridable. + +```bash +# Validation without the gate (fast local loop): +docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld \ + --profile quick-smoke --skip-loki --skip-regression + +# Narrow the rate window to a short profile: +REGRESSION_WINDOW=1m docker/telemetry/workload/run-full-validation.sh \ + --xrpld .build/xrpld --profile quick-smoke + +# Inspect the gate's own output: +jq '.summary' /tmp/xrpld-validation/reports/regression-report.json +jq -r '.metrics[] | select(.regressed) | "\(.key) \(.baseline) -> \(.current) \(.unit)"' \ + /tmp/xrpld-validation/reports/regression-report.json +``` + +#### Refreshing the baseline + +The baseline is a committed file, and moving it is a reviewed change — that PR +review is the audit point for "who moved the performance bar". There is no +automatic promotion from `develop`. + +1. Run the `Telemetry Validation` workflow on the branch. It always captures + timings, so `timings.json` is uploaded as an artifact and the regression + summary is written to the run's Step Summary. +2. If the baseline in the checkout is a placeholder (`"placeholder": true` or an + empty `metrics` object), the Step Summary contains a fenced JSON block under + **"Paste into `baselines/baseline-timings.json`"**, already formatted the way + the file expects (sorted keys, 2-space indent, trailing newline). +3. Open a PR replacing the file contents with that block, dropping the + `placeholder` key. For a refresh of an already-populated baseline, take the + `timings.json` artifact instead and justify the delta in the PR description. + +Never hand-edit `baseline-timings.json` — every entry should trace back to a real +CI run so its variance characteristics are preserved. Details in +`docker/telemetry/workload/baselines/README.md`. + +#### CI workflow + +`.github/workflows/telemetry-validation.yml` runs three jobs — `linux-image-tag` +(reads the CI image tag from the build matrix so this workflow cannot drift onto +a different compiler than the main CI), `build-xrpld` (self-hosted runner, same +container as the main CI, so Conan and ccache hit the shared caches), and +`validate-telemetry` (`ubuntu-latest`, which has Docker). + +- **Triggers**: `workflow_dispatch`, and `push` on `pratik/otel-phase*`, + `feature/otel-*`, `feature/telemetry-*` limited to a `paths` filter covering + the workflow file, `docker/telemetry/**`, and the telemetry sources under + `include/xrpl/telemetry/**`, `src/libxrpl/telemetry/**` and + `src/xrpld/telemetry/**`. There is no cron schedule. +- **Invocation**: `run-full-validation.sh --xrpld --skip-loki`, so the + default `full-validation` profile is used and the Loki checks are skipped. +- **Inputs**: only `run_benchmark` changes behaviour. `rpc_rate`, `rpc_duration`, + `tx_tps` and `tx_duration` are inert, as noted in their descriptions. +- **Results**: reports are uploaded as the `telemetry-validation-reports` + artifact and node logs as `xrpld-node-logs` when validation did not succeed. + Summaries go to the run's Step Summary; the workflow does not comment on PRs. ## Performance Benchmarking @@ -4196,7 +4612,19 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 If benchmarks exceed thresholds: -1. **Reduce sampling**: `sampling_ratio=0.01` (1% of traces) +1. **Reduce trace volume with collector-side tail sampling.** There is no + `sampling_ratio` config key — xrpld's head sampling is a compile-time + constant fixed at 1.0 + ([Telemetry.h:234](../include/xrpl/telemetry/Telemetry.h#L234) + `static constexpr double samplingRatio = 1.0;`), and + [TelemetryConfig.cpp:139](../src/libxrpl/telemetry/TelemetryConfig.cpp#L139) + explicitly parses nothing for it. Volume reduction is a collector decision. + The only policy shipped is a single 0.5% probabilistic `tail_sampling` + processor in `otel-collector-config.grafanacloud.yaml`; the base + `otel-collector-config.yaml` has **no** tail sampling, so a stock local + stack keeps every trace. Where the Cloud policy is in force it sits on the + trace-storage branch only — spanmetrics runs on a separate branch and still + sees 100% of spans, so the derived RED metrics stay exact. 2. **Disable peer tracing**: `trace_peer=0` (highest volume category) 3. **Increase batch delay**: `batch_delay_ms=10000` (less frequent exports) 4. **Reduce queue size**: `max_queue_size=1024` (back-pressure earlier) diff --git a/include/xrpl/beast/insight/OTelCollector.h b/include/xrpl/beast/insight/OTelCollector.h index e043531f99..46d103dc90 100644 --- a/include/xrpl/beast/insight/OTelCollector.h +++ b/include/xrpl/beast/insight/OTelCollector.h @@ -58,7 +58,7 @@ namespace beast::insight { * @code * auto collector = beast::insight::OTelCollector::New( * "http://localhost:4318/v1/metrics", // OTLP/HTTP endpoint - * "xrpld", // metric name prefix + * "xrpld", // logging label only * "node-1", // service.instance.id * "xrpld", // service.name * "mainnet", // xrpl.network.type @@ -105,8 +105,12 @@ public: * * @param endpoint OTLP/HTTP metrics endpoint URL * (e.g. "http://localhost:4318/v1/metrics"). - * @param prefix Prefix prepended to all metric names - * (e.g. "xrpld"). + * @param prefix Label for the collector's startup log line + * (e.g. "xrpld"). Exported metric names are produced + * by formatName(), which lowercases the raw name and + * maps dots and spaces to underscores. The service is + * identified by the `service.name` OTel resource + * attribute. * @param instanceId Unique identifier for this node instance, * emitted as the `service.instance.id` OTel * resource attribute. Defaults to empty string diff --git a/include/xrpl/proto/xrpl.proto b/include/xrpl/proto/xrpl.proto index 468034ad3a..f38c426278 100644 --- a/include/xrpl/proto/xrpl.proto +++ b/include/xrpl/proto/xrpl.proto @@ -95,9 +95,16 @@ message TMPublicKey { // Older peers that do not understand field 1001 will simply ignore it // per protobuf wire-format rules, preserving backwards compatibility. // -// trace_state is reserved for future use (secure tracing pipeline, -// OpenTelemetryPlan/secure-OTel.md). It is currently neither populated -// on inject nor read on extract; consumers must not rely on it. +// trace_state (field 4) is reserved and inert: it is neither populated on +// inject nor read on extract, so consumers must not rely on it. Beyond the +// W3C tracestate use noted on the field below, it is the intended home for +// an authenticated token a receiver could verify before adopting a peer's +// trace context as its parent. Today this message is unauthenticated peer +// input: the receiver only checks that the ids are well formed (16-byte +// trace_id, 8-byte span_id, neither all-zero) and otherwise starts a fresh +// trace, so the ids are a hint, not trusted provenance. An authenticated +// scheme would need a shared verification key, a canonical form to sign, +// and a defined policy for peers that send no token. message TraceContext { optional bytes trace_id = 1; // 16-byte trace identifier optional bytes span_id = 2; // 8-byte parent span identifier diff --git a/include/xrpl/telemetry/TraceContextPropagator.h b/include/xrpl/telemetry/TraceContextPropagator.h index e8d851e53e..d2282665a3 100644 --- a/include/xrpl/telemetry/TraceContextPropagator.h +++ b/include/xrpl/telemetry/TraceContextPropagator.h @@ -101,13 +101,23 @@ injectToProtobuf(opentelemetry::context::Context const& ctx, protocol::TraceCont // Serialize flags proto.set_trace_flags(spanCtx.trace_flags().flags()); - // TODO(observability/secure-OTel): the protobuf TraceContext message - // also carries `trace_state` (field 4), which is currently neither - // populated here nor read by extractFromProtobuf above. The field is - // reserved for the secure tracing pipeline outlined in - // OpenTelemetryPlan/secure-OTel.md, where an authenticated token in - // tracestate will let receivers reject spoofed/poisoned trace context. - // Wire trace_state through inject/extract once the consumer lands. + /** + * TODO: wire `trace_state` (protobuf TraceContext field 4) through + * inject and extract. It is neither written here nor read by + * extractFromProtobuf above, so the field is inert on the wire. + * + * Two uses are intended. One is W3C tracestate vendor-specific + * key-value pairs, for cross-vendor propagation. The other is an + * authenticated token. Today a peer's trace context is + * unauthenticated input: extractFromProtobuf only checks that the ids + * are well formed (16-byte trace_id, 8-byte span_id, neither + * all-zero) before using them as a parent, so the ids are a hint + * rather than trusted provenance. A token the receiver could verify + * would let it decide whether to adopt a peer's context at all. That + * needs a shared verification key, a canonical form to sign, and a + * defined policy for peers that send no token. None of that exists + * yet, which is why the field stays unpopulated. + */ } } // namespace xrpl::telemetry diff --git a/src/libxrpl/basics/Log.cpp b/src/libxrpl/basics/Log.cpp index 46a3829e78..bf46844143 100644 --- a/src/libxrpl/basics/Log.cpp +++ b/src/libxrpl/basics/Log.cpp @@ -302,9 +302,9 @@ Logs::format( } #ifdef XRPL_ENABLE_TELEMETRY - // Inject OTel trace context when an active span exists on this thread. - // Checks the thread-local context value directly to avoid the heap - // allocation that GetSpan() performs on the no-span path. + // Inject OTel trace context when an active, sampled span exists on this + // thread. Checks the thread-local context value directly to avoid the + // heap allocation that GetSpan() performs on the no-span path. { auto context = opentelemetry::context::RuntimeContext::GetCurrent(); auto spanValue = context.GetValue(opentelemetry::trace::kSpanKey); @@ -314,7 +314,18 @@ Logs::format( auto span = opentelemetry::nostd::get< opentelemetry::nostd::shared_ptr>(spanValue); auto spanCtx = span->GetContext(); - if (spanCtx.IsValid()) + // Require the sampled flag as well as a valid context. A dropped + // span still carries its parent's ids, so a valid context does + // not imply the span reaches the backend. An unsampled remote + // parent arrives either because an upstream node propagated + // sampled=0, or because a peer omitted trace_flags entirely and + // it defaults to 0 (TraceContextPropagator, TxTracing, + // ConsensusReceiveTracing). Either way the ParentBasedSampler + // drops the local span, while the tracer still returns a no-op + // span with a valid context. + // Logging those ids would advertise a trace that was never + // exported, leaving the log-to-trace link resolving to nothing. + if (spanCtx.IsValid() && spanCtx.IsSampled()) { // Hex widths of a W3C trace context: 16-byte trace_id and // 8-byte span_id render to 32 and 16 lowercase hex chars. diff --git a/src/libxrpl/beast/insight/OTelCollector.cpp b/src/libxrpl/beast/insight/OTelCollector.cpp index ab17159272..e28dc60ad2 100644 --- a/src/libxrpl/beast/insight/OTelCollector.cpp +++ b/src/libxrpl/beast/insight/OTelCollector.cpp @@ -5,7 +5,7 @@ * Compiled only when XRPL_ENABLE_TELEMETRY is defined (via CMake * telemetry=ON). Maps beast::insight instruments to OTel SDK instruments * created on the GLOBAL Meter published by the telemetry module. This class - * is a legacy shim: it no longer owns an export pipeline. The MeterProvider, + * is an adapter only: it owns no export pipeline. The MeterProvider, * PeriodicExportingMetricReader, OTLP exporter and histogram view all live in * xrpl::telemetry::Telemetry. * @@ -134,8 +134,8 @@ class OTelCounterImpl : public CounterImpl public: /** * @param name Export-ready metric name, already run through - * formatName() by the collector: prefix prepended and - * dots replaced with underscores (e.g. "rpc_size"). + * formatName() by the collector: lowercase, with `.` and + * ` ` mapped to `_` (e.g. "rpc_size"). * @param meter OTel Meter used to create the counter instrument. */ OTelCounterImpl( @@ -178,8 +178,8 @@ class OTelEventImpl : public EventImpl public: /** * @param name Export-ready metric name, already run through - * formatName() by the collector: prefix prepended and - * dots replaced with underscores (e.g. "rpc_size"). + * formatName() by the collector: lowercase, with `.` and + * ` ` mapped to `_` (e.g. "rpc_size"). * @param meter OTel Meter used to create the histogram instrument. */ OTelEventImpl( @@ -227,8 +227,8 @@ class OTelGaugeImpl : public GaugeImpl public: /** * @param name Export-ready metric name, already run through - * formatName() by the collector: prefix prepended - * and dots replaced with underscores. + * formatName() by the collector: lowercase, with `.` + * and ` ` mapped to `_`. * @param meter OTel Meter used to create the observable gauge. * @param collector Owning collector, used to invoke hooks before reads. */ @@ -310,8 +310,8 @@ class OTelMeterImpl : public MeterImpl public: /** * @param name Export-ready metric name, already run through - * formatName() by the collector: prefix prepended and - * dots replaced with underscores (e.g. "rpc_size"). + * formatName() by the collector: lowercase, with `.` and + * ` ` mapped to `_` (e.g. "rpc_size"). * @param meter OTel Meter used to create the counter instrument. */ OTelMeterImpl( @@ -340,7 +340,7 @@ private: //------------------------------------------------------------------------------ /** - * @brief Main OTel Collector implementation (legacy shim). + * @brief Main OTel Collector implementation (adapter over the global Meter). * * Obtains its Meter from the GLOBAL MeterProvider owned and published by the * telemetry module (xrpl::telemetry::Telemetry), rather than building its own @@ -380,8 +380,11 @@ private: * Caveats: * - Observable gauge callbacks run on the SDK's internal thread. Hook * handlers must be thread-safe. - * - Metric names are formed as "prefix_name" with dots replaced by - * underscores to match StatsD->Prometheus naming conventions. + * - Metric names carry NO prefix. formatName() only lowercases the raw + * name and turns dots and spaces into underscores, to match + * StatsD->Prometheus naming conventions. The service is identified by + * the OTel resource (service.name), so prefix_ is kept for logging + * only and never affects an exported name. * - The OTel Prometheus exporter appends "_total" to counters. The * metric names we register do NOT include this suffix — Prometheus * adds it automatically. @@ -402,11 +405,14 @@ public: /** * @brief Construct the OTel collector over the global MeterProvider. * - * @param endpoint OTLP/HTTP metrics endpoint URL. Informational only: - * the global telemetry pipeline is authoritative for - * the actual export endpoint. Retained for logging and - * back-compat with the New() signature. - * @param prefix Prefix for all metric names. + * @param endpoint OTLP/HTTP metrics endpoint URL, recorded in the + * collector's startup log line. Export uses the + * endpoint configured on the global telemetry + * pipeline. + * @param prefix Label for the collector's startup log line + * (e.g. "xrpld"). Exported metric names come from + * formatName(); the service is identified by the + * service.name resource attribute. * @param instanceId Value for the service.instance.id resource attribute. * When empty, the attribute is omitted. * @param serviceName Value for the service.name resource attribute. @@ -498,10 +504,12 @@ public: /** @} */ /** - * @brief Format a metric name with the configured prefix. + * @brief Format a raw metric name for export. * - * Replaces dots with underscores to match StatsD->Prometheus naming. - * Example: prefix="xrpld", name="LedgerMaster.Validated_Ledger_Age" + * Lowercases the name and replaces dots and spaces with underscores to + * match StatsD->Prometheus naming. Adds NO prefix: the service is + * identified by the OTel resource (service.name). + * Example: name="LedgerMaster.Validated_Ledger_Age" * -> "ledgermaster_validated_ledger_age" * * @param name Raw metric name from beast::insight callers. @@ -517,7 +525,8 @@ private: Journal journal_; /** - * Prefix for all metric names (e.g., "xrpld"). + * Configured metric-name prefix (e.g., "xrpld"). Log-only: it is + * echoed in the startup log line and never applied to a metric name. */ std::string prefix_; @@ -708,17 +717,17 @@ OTelCollectorImp::OTelCollectorImp( Journal journal) : journal_(journal), prefix_(std::move(prefix)) { - // instanceId/serviceName/networkType are retained on the New() signature - // for back-compat but no longer used here: the telemetry module owns the - // resource attributes for the shared metrics pipeline. + // instanceId/serviceName/networkType are accepted but unused here: the + // telemetry module owns the resource attributes for the shared metrics + // pipeline, so setting them from this collector would have no effect. (void)instanceId; (void)serviceName; (void)networkType; if (journal_.info()) { - // endpoint is informational: the global telemetry pipeline owns the - // real exporter. It is logged here for back-compat and diagnostics. + // endpoint is logged for diagnostics only: the global telemetry + // pipeline owns the exporter that actually sends the metrics. journal_.info() << "OTelCollector starting: endpoint=" << endpoint << " prefix=" << prefix_; } @@ -846,9 +855,9 @@ OTelCollectorImp::removeGauge(OTelGaugeImpl* gauge) std::string OTelCollectorImp::formatName(std::string const& name) { - // Produce a clean, lowercase, Prometheus-compatible metric name. - // No prefix — the OTel resource (service.name) identifies the service. - // Dots and spaces become underscores; everything lowercased. + // Produce a lowercase, Prometheus-compatible metric name: dots and + // spaces become underscores. Service identity travels in the + // service.name resource attribute, not in the metric name. std::string result; result.reserve(name.size()); for (char const c : name) diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp index d36e7594c6..a2c13ed463 100644 --- a/src/tests/libxrpl/telemetry/MetricMacros.cpp +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -589,9 +589,9 @@ TEST(MetricMacros, observable_gauge_register_reports_current_value) FakeApp app; wire(app, /*enabled=*/true); - // Own the state exactly as a real caller would (Use Case 5 in the - // design doc) -- the macro's callback reads through this atomic on - // every collection tick, it does not own the value itself. + // Own the state exactly as a real caller would -- the macro's callback + // reads through this atomic on every collection tick, it does not own + // the value itself. std::atomic queueDepth{0}; XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER( app, @@ -599,11 +599,11 @@ TEST(MetricMacros, observable_gauge_register_reports_current_value) "Test observable gauge for macro unit test", [&queueDepth] { return queueDepth.load(); }); - // There is no application-level read-back API (Use Case 4) -- this - // test can only prove registration doesn't crash and that meter() was - // consulted to create the observable instrument. It does NOT assert the - // observed value reaches Prometheus; that is Task 3b's docker-harness - // job, not this hermetic unit test. + // There is no application-level read-back API -- this test can only + // prove registration doesn't crash and that meter() was consulted to + // create the observable instrument. It does NOT assert the observed + // value reaches Prometheus; that is the docker-harness integration + // test's job, not this hermetic unit test. queueDepth.store(42); EXPECT_EQ(app.registry().meterCalls(), 1); } diff --git a/src/tests/libxrpl/telemetry/TraceContextPropagator.cpp b/src/tests/libxrpl/telemetry/TraceContextPropagator.cpp index fdc6c443cb..aa0bbf8903 100644 --- a/src/tests/libxrpl/telemetry/TraceContextPropagator.cpp +++ b/src/tests/libxrpl/telemetry/TraceContextPropagator.cpp @@ -17,8 +17,6 @@ #include #include -#include - #include #include diff --git a/src/xrpld/app/consensus/RCLConsensus.cpp b/src/xrpld/app/consensus/RCLConsensus.cpp index 2098352a4a..01ed1e2b35 100644 --- a/src/xrpld/app/consensus/RCLConsensus.cpp +++ b/src/xrpld/app/consensus/RCLConsensus.cpp @@ -277,9 +277,9 @@ RCLConsensus::Adaptor::propose(RCLCxPeerPos::Proposal const& proposal) app_.getHashRouter().addSuppression(suppression); - // Inject the current thread's active span context (e.g. the - // consensus round span from Phase 4) so receiving peers can link - // their proposal.receive span as a child of this trace. + // Inject the current thread's active span context (e.g. the consensus + // round span) so receiving peers can link their proposal.receive span + // as a child of this trace. telemetry::SpanGuard::injectCurrentContextToProtobuf(*prop.mutable_trace_context()); app_.getOverlay().broadcast(prop); @@ -765,7 +765,7 @@ RCLConsensus::Adaptor::doAccept( // Record ledger close for OTel dashboard parity counter. Uses the // call-site macro (see MetricMacros.h) rather than a MetricsRegistry - // member -- proof-of-concept for tasks/metric-macro-plan.md. + // member. XRPL_METRIC_COUNTER_INC(app_, "ledgers_closed_total", "Total ledgers closed by consensus"); //------------------------------------------------------------------------- diff --git a/src/xrpld/app/consensus/RCLConsensus.h b/src/xrpld/app/consensus/RCLConsensus.h index 1ada9e7ee9..a7057e2f3a 100644 --- a/src/xrpld/app/consensus/RCLConsensus.h +++ b/src/xrpld/app/consensus/RCLConsensus.h @@ -127,9 +127,9 @@ class RCLConsensus * * Captured in makeAcceptSpan() and consumed by createValidationSpan() * on the jtACCEPT worker thread so the validation.send span can be - * follows-from linked to consensus.accept (matching the design doc - * and span hierarchy diagram). Reset on each startRoundTracing() - * to prevent a stale prior-round context from being linked. + * follows-from linked to consensus.accept. Reset on each + * startRoundTracing() to prevent a stale prior-round context from + * being linked. * * Thread safety: same model as roundSpanContext_. The write in * makeAcceptSpan happens on the main consensus thread under diff --git a/src/xrpld/app/ledger/detail/LedgerSpanNames.h b/src/xrpld/app/ledger/detail/LedgerSpanNames.h index 9449593e44..c34cb8b47b 100644 --- a/src/xrpld/app/ledger/detail/LedgerSpanNames.h +++ b/src/xrpld/app/ledger/detail/LedgerSpanNames.h @@ -177,6 +177,12 @@ inline constexpr auto abandoned = makeStr("abandoned"); */ inline constexpr auto timeout = makeStr("timeout"); +/** + * Set when the acquisition is abandoned before it finishes, i.e. the + * InboundLedger is destroyed while !isDone(). Distinct from `failed`, which + * means the fetch ran to its retry limit and gave up. + */ +inline constexpr auto aborted = makeStr("aborted"); /** * ledger.acquire reason values (mirror InboundLedger::Reason). */ diff --git a/src/xrpld/perflog/detail/PerfLogImp.cpp b/src/xrpld/perflog/detail/PerfLogImp.cpp index f43464dd1d..b3cd723872 100644 --- a/src/xrpld/perflog/detail/PerfLogImp.cpp +++ b/src/xrpld/perflog/detail/PerfLogImp.cpp @@ -331,17 +331,21 @@ PerfLogImp::rpcStart(std::string const& method, std::uint64_t const requestId) std::scoped_lock const lock(counter->second.mutex); ++counter->second.value.started; } - std::scoped_lock const lock(counters_.methodsMutex); - counters_.methods[requestId] = {counter->first.c_str(), steady_clock::now()}; + { + std::scoped_lock const lock(counters_.methodsMutex); + counters_.methods[requestId] = {counter->first.c_str(), steady_clock::now()}; + } - // Task 9.4: Record RPC start in OTel metrics pipeline. + // Record RPC start in OTel metrics pipeline. Recorded after the locks + // above are released: the OTel call path allocates and takes locks + // inside the SDK, so holding methodsMutex across it would widen a + // process-wide critical section for no reason. Mirrors rpcEnd(). if (auto* mr = app_.getMetricsRegistry()) mr->recordRpcStarted(method); - // Proof-of-concept for tasks/metric-macro-plan.md Use Case 2: a value - // that must be able to decrease (UpDownCounter), added at its call - // site with no MetricsRegistry member/init-line/method. Paired with the - // matching -1 in rpcEnd(). Runs on the same path as recordRpcStarted + // A value that must be able to decrease (UpDownCounter), added at its + // call site with no MetricsRegistry member/init-line/method. Paired with + // the matching -1 in rpcEnd(). Runs on the same path as recordRpcStarted // above, i.e. only after a methods-map entry exists for this request. XRPL_METRIC_UPDOWN_ADD(app_, "rpc_in_flight_requests", "RPC requests currently executing", 1); } @@ -392,9 +396,9 @@ PerfLogImp::rpcEnd(std::string const& method, std::uint64_t const requestId, boo counter->second.value.duration += durationUs; } - // Task 9.4: Record RPC completion in OTel metrics pipeline. - // Mirrors the rpcStart() instrumentation so the finished/errored - // counters and duration histogram advance with every call. + // Record RPC completion in OTel metrics pipeline. Mirrors the + // rpcStart() instrumentation so the finished/errored counters and + // duration histogram advance with every call. if (auto* mr = app_.getMetricsRegistry()) { if (finish) @@ -424,10 +428,13 @@ PerfLogImp::jobQueue(JobType const type, std::string const& name) return; // LCOV_EXCL_STOP } - std::scoped_lock const lock(counter->second.mutex); - ++counter->second.value.queued; + { + std::scoped_lock const lock(counter->second.mutex); + ++counter->second.value.queued; + } - // Task 9.5: Record job enqueue in OTel metrics pipeline. + // Record job enqueue in OTel metrics pipeline, after the lock above is + // released so the SDK's work stays outside the critical section. if (auto* mr = app_.getMetricsRegistry()) mr->recordJobQueued(JobTypes::name(type), name); } @@ -454,11 +461,16 @@ PerfLogImp::jobStart( ++counter->second.value.started; counter->second.value.queuedDuration += dur; } - std::scoped_lock const lock(counters_.jobsMutex); - if (instance >= 0 && instance < counters_.jobs.size()) - counters_.jobs[instance] = {type, startTime}; + { + std::scoped_lock const lock(counters_.jobsMutex); + if (instance >= 0 && instance < counters_.jobs.size()) + counters_.jobs[instance] = {type, startTime}; + } - // Task 9.5: Record job start in OTel metrics pipeline. + // Record job start in OTel metrics pipeline, after the locks above are + // released. jobsMutex is process-wide and taken by every worker thread + // on every job, so the SDK's allocation and internal locking must not + // run inside it. if (auto* mr = app_.getMetricsRegistry()) mr->recordJobStarted(JobTypes::name(type), name, dur.count()); } @@ -480,11 +492,14 @@ PerfLogImp::jobFinish(JobType const type, std::string const& name, microseconds ++counter->second.value.finished; counter->second.value.runningDuration += dur; } - std::scoped_lock const lock(counters_.jobsMutex); - if (instance >= 0 && instance < counters_.jobs.size()) - counters_.jobs[instance] = {JtInvalid, steady_time_point()}; + { + std::scoped_lock const lock(counters_.jobsMutex); + if (instance >= 0 && instance < counters_.jobs.size()) + counters_.jobs[instance] = {JtInvalid, steady_time_point()}; + } - // Task 9.5: Record job finish in OTel metrics pipeline. + // Record job finish in OTel metrics pipeline, after the locks above + // are released, for the same reason as jobStart(). if (auto* mr = app_.getMetricsRegistry()) mr->recordJobFinished(JobTypes::name(type), name, dur.count()); } diff --git a/src/xrpld/telemetry/ConsensusReceiveTracing.h b/src/xrpld/telemetry/ConsensusReceiveTracing.h index 03974c5473..0a1a4458dc 100644 --- a/src/xrpld/telemetry/ConsensusReceiveTracing.h +++ b/src/xrpld/telemetry/ConsensusReceiveTracing.h @@ -34,7 +34,8 @@ * * @note Span names come from the canonical constants in * ConsensusSpanNames.h (consensus::span::proposalReceive / - * validationReceive) so they stay in sync with the rest of Phase 4. + * validationReceive) so they stay in sync with the rest of the + * consensus tracing surface. */ #include diff --git a/src/xrpld/telemetry/MetricMacros.h b/src/xrpld/telemetry/MetricMacros.h index 0390a44167..1b008cd6c5 100644 --- a/src/xrpld/telemetry/MetricMacros.h +++ b/src/xrpld/telemetry/MetricMacros.h @@ -83,27 +83,26 @@ * @note A histogram whose values can exceed ~10,000 units (e.g. a * microsecond duration beyond 10ms) needs an explicit-bucket View, which * OTel can only register at MeterProvider construction time -- this - * cannot be done from a call site. See Limitation 2. Register such a - * view in MetricsRegistry::initExporterAndProvider() as today; the + * cannot be done from a call site. Register such a view in + * MetricsRegistry::initExporterAndProvider() as today; the * histogram-record call itself can still use the macro. * * @note Only call the SYNCHRONOUS macros (Counter/UpDownCounter/ * Histogram/Gauge) from code that runs AFTER MetricsRegistry::start() has * completed (RPC handlers, job callbacks, consensus rounds, tx apply, peer - * message handlers). See Limitation 1. + * message handlers). * * @note The OBSERVABLE registration macros are the opposite: call them * EAGERLY, exactly once, from constructor/init code -- never from a hot * path. Repeated calls at the same call site register a NEW callback * each time (no create-once caching, unlike the synchronous macros), - * which leaks callbacks. See Limitation 3. + * which leaks callbacks. * * @note There is no way to read back a synchronous instrument's current * accumulated value from application code -- the OTel API is * write-only/push-based by design. If your logic needs both to record a * metric AND read its running value, keep your own state (std::atomic or - * similar) and separately feed OTel via these macros. See "Use Case 4" in - * tasks/metric-macro-plan.md. + * similar) and separately feed OTel via these macros. */ // On Windows, OTel's spin_lock_mutex.h (transitively included from @@ -204,11 +203,11 @@ } while (false) // UpDownCounter: like COUNTER_ADD, but the underlying instrument permits a -// negative amount (Use Case 2 -- e.g. in-flight request count, +1 on start -// / -1 on finish from two different points in the same or different call -// sites). A plain Counter's Add() must never see a negative value per the -// OTel API contract; use this macro, not COUNTER_ADD, whenever the value -// can decrease. +// negative amount (e.g. in-flight request count, +1 on start / -1 on +// finish from two different points in the same or different call sites). +// A plain Counter's Add() must never see a negative value per the OTel +// API contract; use this macro, not COUNTER_ADD, whenever the value can +// decrease. #define XRPL_METRIC_UPDOWN_ADD(app, name, description, amount) \ do \ { \ @@ -350,15 +349,15 @@ #endif // OPENTELEMETRY_ABI_VERSION_NO >= 2 // ----------------------------------------------------------------- -// Observable/async instrument registration (Use Case 5). Unlike the -// synchronous macros above, these do NOT lazily create-on-first-call -- -// they register a callback with the SDK immediately, at the call site, -// the moment the macro executes. Callers MUST invoke this during +// Observable/async instrument registration. Unlike the synchronous +// macros above, these do NOT lazily create-on-first-call -- they +// register a callback with the SDK immediately, at the call site, the +// moment the macro executes. Callers MUST invoke this during // construction/init, before the server is fully live (same timing rule // MetricsRegistry::registerAsyncGauges() already follows for its own -// gauges -- see Limitation 3). Calling it from a hot-path function -// instead of an init path re-registers a new callback on every call, -// which leaks callbacks and is NOT what this macro is for. +// gauges). Calling it from a hot-path function instead of an init path +// re-registers a new callback on every call, which leaks callbacks and +// is NOT what this macro is for. // // The callable is captured in a heap-allocated std::function, and its // address is passed as the `void* state` to AddCallback (whose signature, diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/MetricsRegistry.cpp index 2b0e21ef8d..1e2f9ce6dd 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/MetricsRegistry.cpp @@ -24,6 +24,23 @@ #ifdef XRPL_ENABLE_TELEMETRY +// The app and overlay includes below are why +// .github/scripts/levelization/results/loops.txt records +// `xrpld.app <-> xrpld.telemetry` and `xrpld.overlay <-> xrpld.telemetry`, where +// ordering.txt previously had telemetry strictly below both. The observable +// gauges are pull-model: their callbacks sample live state when the reader +// thread fires, so they need the concrete types to call getJqTransOverflow(), +// size(), getPeerDisconnectCharges(), foreach() and txMetrics(). +// +// The cycle is confined to this translation unit. No telemetry header includes +// app or overlay (MetricsRegistry.h forward-declares what it needs and takes a +// ServiceRegistry&), and all of src/xrpld builds into a single CMake target, so +// there is no header cycle and no link cycle to break. +// +// Inverting it properly means declaring a metrics-source interface below overlay +// and implementing it there, which is deliberately left as follow-up rather than +// widening this change. Note loops.txt is generated: it can only change as a +// consequence of changing these includes, never by editing the baseline. #include #include #include @@ -451,7 +468,7 @@ MetricsRegistry::initSyncInstruments() jobRunningDurationHistogram_ = meter_->CreateDoubleHistogram(kJobRunningDurationUs, "Job execution time in microseconds"); - // --- External dashboard parity counters (Task 7.14) --- + // --- External dashboard parity counters --- ledgersClosedCounter_ = meter_->CreateUInt64Counter("ledgers_closed_total", "Total ledgers closed by consensus"); validationsSentCounter_ = meter_->CreateUInt64Counter( @@ -512,7 +529,7 @@ MetricsRegistry::stop() } // ----------------------------------------------------------------- -// Synchronous instrument recording — RPC metrics (Task 9.4) +// Synchronous instrument recording — RPC metrics // ----------------------------------------------------------------- void @@ -571,7 +588,7 @@ MetricsRegistry::recordRpcErrored(std::string_view method, std::int64_t duration } // ----------------------------------------------------------------- -// Synchronous instrument recording — Job Queue metrics (Task 9.5) +// Synchronous instrument recording — Job Queue metrics // ----------------------------------------------------------------- void @@ -651,7 +668,7 @@ MetricsRegistry::recordJobFinished( } // ----------------------------------------------------------------- -// Observable gauge callbacks (Tasks 9.1, 9.2, 9.3, 9.6, 9.7) +// Observable gauge callbacks // ----------------------------------------------------------------- #ifdef XRPL_ENABLE_TELEMETRY @@ -731,7 +748,7 @@ MetricsRegistry::registerJqTransOverflowCounter() void MetricsRegistry::registerCacheHitRateGauge() { - // --- Task 9.2: Cache hit rate and size gauges --- + // --- Cache hit rate and size gauges --- cacheHitRateGauge_ = meter_->CreateDoubleObservableGauge("cache_metrics", "Cache hit rates and sizes"); cacheHitRateGauge_->AddCallback( @@ -802,7 +819,7 @@ MetricsRegistry::registerCacheHitRateGauge() void MetricsRegistry::registerTxqGauge() { - // --- Task 9.3: TxQ metrics gauges --- + // --- TxQ metrics gauges --- txqGauge_ = meter_->CreateDoubleObservableGauge("txq_metrics", "Transaction queue metrics"); txqGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { @@ -849,7 +866,7 @@ MetricsRegistry::registerTxqGauge() void MetricsRegistry::registerObjectCountGauge() { - // --- Task 9.6: Counted object instance gauges --- + // --- Counted object instance gauges --- objectCountGauge_ = meter_->CreateInt64ObservableGauge( "object_count", "Live instance counts for key internal object types"); objectCountGauge_->AddCallback( @@ -881,7 +898,7 @@ MetricsRegistry::registerObjectCountGauge() void MetricsRegistry::registerLoadFactorGauge() { - // --- Task 9.7: Load factor breakdown gauges --- + // --- Load factor breakdown gauges --- loadFactorGauge_ = meter_->CreateDoubleObservableGauge("load_factor_metrics", "Fee load factor breakdown"); loadFactorGauge_->AddCallback( @@ -1045,7 +1062,7 @@ MetricsRegistry::observeReadQueue(node_store::Database& db, ObserveFn const& obs void MetricsRegistry::registerNodeStoreGauge() { - // --- Task 9.1: NodeStore I/O gauges --- + // --- NodeStore I/O gauges --- // The cumulative counters (reads, writes, bytes) are also exposed here // as observable gauges. This avoids adding an xrpld dependency into the // libxrpl nodestore code — the MetricsRegistry reads the existing atomic @@ -1157,7 +1174,7 @@ MetricsRegistry::registerRotationStateGauge() void MetricsRegistry::registerServerInfoGauge() { - // --- Task 9.7a: Server info gauges --- + // --- Server info gauges --- serverInfoGauge_ = meter_->CreateInt64ObservableGauge(metric::serverInfo, "Server-level health metrics"); serverInfoGauge_->AddCallback( @@ -1242,7 +1259,7 @@ MetricsRegistry::registerServerInfoGauge() void MetricsRegistry::registerBuildInfoGauge() { - // --- Task 9.7b: Build info gauge --- + // --- Build info gauge --- buildInfoGauge_ = meter_->CreateInt64ObservableGauge("build_info", "Build version information"); buildInfoGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* /* state */) { @@ -1262,7 +1279,7 @@ MetricsRegistry::registerBuildInfoGauge() void MetricsRegistry::registerCompleteLedgersGauge() { - // --- Task 9.7c: Complete ledgers range gauge --- + // --- Complete ledgers range gauge --- completeLedgersGauge_ = meter_->CreateInt64ObservableGauge( "complete_ledgers", "Complete ledger range start/end pairs"); completeLedgersGauge_->AddCallback( @@ -1321,7 +1338,7 @@ MetricsRegistry::registerCompleteLedgersGauge() void MetricsRegistry::registerDbMetricsGauge() { - // --- Task 9.7d: Database size and fetch rate gauges --- + // --- Database size and fetch rate gauges --- dbMetricsGauge_ = meter_->CreateInt64ObservableGauge("db_metrics", "Database storage sizes and fetch rates"); dbMetricsGauge_->AddCallback( @@ -1360,7 +1377,7 @@ MetricsRegistry::registerDbMetricsGauge() void MetricsRegistry::registerValidatorHealthGauge() { - // --- Task 7.9: Validator health gauges --- + // --- Validator health gauges --- validatorHealthGauge_ = meter_->CreateDoubleObservableGauge("validator_health", "Validator health indicators"); validatorHealthGauge_->AddCallback( @@ -1407,7 +1424,7 @@ MetricsRegistry::registerValidatorHealthGauge() void MetricsRegistry::registerPeerQualityGauge() { - // --- Task 7.10: Peer quality gauges --- + // --- Peer quality gauges --- // Uses Peer::json() to read latency and version since those accessors // are not on the abstract Peer interface (they live on PeerImp). peerQualityGauge_ = @@ -1561,7 +1578,7 @@ MetricsRegistry::registerReduceRelayGauge() void MetricsRegistry::registerLedgerEconomyGauge() { - // --- Task 7.11: Ledger economy gauges --- + // --- Ledger economy gauges --- ledgerEconomyGauge_ = meter_->CreateDoubleObservableGauge( metric::ledgerEconomy, "Ledger fee and economy metrics"); ledgerEconomyGauge_->AddCallback( @@ -1626,7 +1643,7 @@ MetricsRegistry::registerLedgerEconomyGauge() void MetricsRegistry::registerStateTrackingGauge() { - // --- Task 7.12: State tracking gauges --- + // --- State tracking gauges --- stateTrackingGauge_ = meter_->CreateDoubleObservableGauge(metric::stateTracking, "Node state and mode tracking"); stateTrackingGauge_->AddCallback( @@ -1680,7 +1697,7 @@ MetricsRegistry::registerStateTrackingGauge() void MetricsRegistry::registerStorageDetailGauge() { - // --- Task 7.13: Storage detail gauges --- + // --- Storage detail gauges --- // Reports the cumulative payload bytes handed to the NodeStore. See the // note at the observe() call below: this is logical bytes stored, not // on-disk file size, because no accessor for the latter exists. The label @@ -1732,7 +1749,7 @@ MetricsRegistry::registerStorageDetailGauge() void MetricsRegistry::registerValidationAgreementGauge() { - // --- Task 7.15: Validation agreement gauges --- + // --- Validation agreement gauges --- // Reports rolling-window agreement percentages and counts from // ValidationTracker. reconcile() is called at the start of the // callback so that pending ledger events are resolved before the @@ -2383,7 +2400,7 @@ MetricsRegistry::registerLedgerQuorumPublishGauge() #endif // XRPL_ENABLE_TELEMETRY // ----------------------------------------------------------------- -// External dashboard parity counter increments (Task 7.14) +// External dashboard parity counter increments // ----------------------------------------------------------------- void diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/src/xrpld/telemetry/MetricsRegistry.h index bd44796ff5..fce4a56d1f 100644 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ b/src/xrpld/telemetry/MetricsRegistry.h @@ -241,8 +241,8 @@ namespace telemetry { * edit needed. Fall back to a dedicated member + init line + record * method (the pattern below) only when the metric needs to be read * back by other code (e.g. ValidationTracker-style accumulation) or - * needs a custom histogram bucket View (see MetricMacros.h Limitation - * 2 in tasks/metric-macro-plan.md). + * needs a custom histogram bucket View (see the histogram note in + * MetricMacros.h). * - Adding a new OBSERVABLE gauge still requires eager central * registration -- pull-model instruments cannot be lazily created. */ @@ -585,7 +585,7 @@ public: std::int64_t runningDurUs); // ----------------------------------------------------------------- - // External dashboard parity counters (Tasks 7.9-7.14) + // External dashboard parity counters // ----------------------------------------------------------------- /** @@ -813,7 +813,7 @@ private: */ opentelemetry::nostd::unique_ptr> rpcErroredCounter_; /** - * Histogram: rpc_method_duration_us{method=""} + * Histogram: rpc_method_us{method=""} */ opentelemetry::nostd::unique_ptr> rpcDurationHistogram_; @@ -834,12 +834,12 @@ private: */ opentelemetry::nostd::unique_ptr> jobFinishedCounter_; /** - * Histogram: job_queued_duration_us{job_type="",handler=""} + * Histogram: job_queued_us{job_type="",handler=""} */ opentelemetry::nostd::unique_ptr> jobQueuedDurationHistogram_; /** - * Histogram: job_running_duration_us{job_type="",handler=""} + * Histogram: job_running_us{job_type="",handler=""} */ opentelemetry::nostd::unique_ptr> jobRunningDurationHistogram_; @@ -959,7 +959,7 @@ private: */ opentelemetry::nostd::shared_ptr dbMetricsGauge_; - // --- External dashboard parity gauges (Tasks 7.9-7.13) --- + // --- External dashboard parity gauges --- /** * Observable gauge for validator health indicators (amendment blocked, * UNL blocked, quorum, UNL expiry). @@ -1002,7 +1002,7 @@ private: opentelemetry::nostd::shared_ptr validationAgreementGauge_; - // --- External dashboard parity counters (Task 7.14) --- + // --- External dashboard parity counters --- /** * Counter: ledgers_closed_total — incremented each consensus round. */ @@ -1095,15 +1095,15 @@ private: void registerJqTransOverflowCounter(); // gap-fill: overlay overflow total void - registerCacheHitRateGauge(); // Task 9.2 + registerCacheHitRateGauge(); void - registerTxqGauge(); // Task 9.3 + registerTxqGauge(); void - registerObjectCountGauge(); // Task 9.6 + registerObjectCountGauge(); void - registerLoadFactorGauge(); // Task 9.7 + registerLoadFactorGauge(); void - registerNodeStoreGauge(); // Task 9.1 + registerNodeStoreGauge(); // The four nodestore_state helpers and their ObserveFn sink are public // (above), so a test can drive each one with a recording sink and assert @@ -1113,27 +1113,27 @@ private: void registerRotationStateGauge(); // Sync diagnostics: online_delete rotation void - registerServerInfoGauge(); // Task 9.7a + registerServerInfoGauge(); void - registerBuildInfoGauge(); // Task 9.7b + registerBuildInfoGauge(); void - registerCompleteLedgersGauge(); // Task 9.7c + registerCompleteLedgersGauge(); void - registerDbMetricsGauge(); // Task 9.7d + registerDbMetricsGauge(); void - registerValidatorHealthGauge(); // Task 7.9 + registerValidatorHealthGauge(); void - registerPeerQualityGauge(); // Task 7.10 + registerPeerQualityGauge(); void registerReduceRelayGauge(); // Reduce-relay efficiency void - registerLedgerEconomyGauge(); // Task 7.11 + registerLedgerEconomyGauge(); void - registerStateTrackingGauge(); // Task 7.12 + registerStateTrackingGauge(); void - registerStorageDetailGauge(); // Task 7.13 + registerStorageDetailGauge(); void - registerValidationAgreementGauge(); // Task 7.15 + registerValidationAgreementGauge(); void registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total diff --git a/tasks/fix-validation-checks.md b/tasks/fix-validation-checks.md deleted file mode 100644 index ea33fd9223..0000000000 --- a/tasks/fix-validation-checks.md +++ /dev/null @@ -1,169 +0,0 @@ -# Fix Telemetry Validation Checks - -## Context - -The CI pipeline infrastructure is fully operational (build + deploy + run). However, -the `validate_telemetry.py` validation suite fails 35 checks due to mismatches between -what the validation expects and what the telemetry stack actually produces. These fall -into 4 categories. - -CI run: https://github.com/XRPLF/rippled/actions/runs/23026466191 - ---- - -## Category 1: StatsD Metrics — 0 Series (25 failures) - -**Symptoms:** - -``` -[FAIL] metric.statsd_gauges.xrpld_LedgerMaster_Validated_Ledger_Age: 0 series -[FAIL] metric.statsd_counters.xrpld_rpc_requests: 0 series -[FAIL] metric.statsd_histograms.xrpld_rpc_time: 0 series -[FAIL] metric.overlay_traffic.xrpld_total_Bytes_In: 0 series -[FAIL] metric.phase9_nodestore.xrpld_nodestore_reads_total: 0 series -... (25 total) -``` - -**Root Cause:** Two issues compounding: - -1. **StatsD receiver is commented out** in `otel-collector-config.yaml` (lines 39-54). - The collector config was updated to expect native OTLP metrics from beast::insight - (comment: "StatsD UDP port removed — beast::insight now uses native OTLP"), but - the validation harness configures xrpld nodes with `server=statsd`. - -2. **Metric name mismatch:** The `expected_metrics.json` expects StatsD-style metric - names (e.g., `xrpld_LedgerMaster_Validated_Ledger_Age`). When using `server=otel`, - beast::insight emits OTLP metrics which may have different names/structure. - -**Fix Options (pick one):** - -- **Option A (recommended):** Change the node config in `run-full-validation.sh` from - `server=statsd` to `server=otel` (line 255), remove the `address=127.0.0.1:8125` line, - then update `expected_metrics.json` with the actual OTLP metric names. This aligns with - the collector config's OTLP-first design and avoids re-enabling the StatsD receiver. - -- **Option B:** Uncomment the StatsD receiver in `otel-collector-config.yaml`, add - `statsd` to the metrics pipeline receivers list, and keep node config as `server=statsd`. - Simpler but goes against the migration to native OTLP. - -**Investigation needed for Option A:** - -- Run xrpld locally with `server=otel`, query Prometheus, and capture the actual OTLP - metric names to update `expected_metrics.json`. - -**Files to modify:** - -- `docker/telemetry/workload/run-full-validation.sh` — change `[insight]` section -- `docker/telemetry/workload/expected_metrics.json` — update metric names for OTLP -- `docker/telemetry/workload/validate_telemetry.py` — may need metric query adjustments - ---- - -## Category 2: Missing Spans — tx.process, tx.receive (2 failures) - -**Symptoms:** - -``` -[FAIL] span.tx.process: tx.process: 0 traces (expected > 0) -[FAIL] span.tx.receive: tx.receive: 0 traces (expected > 0) -``` - -**Root Cause:** The span names exist in the code: - -- `src/xrpld/app/misc/NetworkOPs.cpp:1228` — `XRPL_TRACE_TX("tx.process")` -- `src/xrpld/overlay/detail/PeerImp.cpp:1273` — `XRPL_TRACE_TX("tx.receive")` - -Likely causes (investigate in order): - -1. **Batch delay:** The 2-second batch delay (`batch_delay_ms=2000`) plus 30s propagation - wait may not be enough if these spans are created late in the workload. -2. **Code path not triggered:** `tx.process` fires in `NetworkOPs::processTransaction()`. - The tx_submitter submits via RPC `submit` command which calls this path. But if the - transactions fail validation before reaching `processTransaction()`, no span is emitted. -3. **Span naming mismatch:** The validation queries Tempo for exact operation name - `tx.process`. Verify Tempo stores the span with this exact name. - -**Investigation:** - -- Check the tx_submitter output in CI logs — are transactions actually succeeding? -- Query Tempo API locally for all span names to see what's actually emitted. - -**Files to modify:** - -- Possibly `docker/telemetry/workload/validate_telemetry.py` — adjust timing/queries -- Possibly `docker/telemetry/workload/run-full-validation.sh` — increase propagation wait - ---- - -## Category 3: Span Hierarchy — rpc.request -> rpc.process (1 failure) - -**Symptoms:** - -``` -[FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces -``` - -**Root Cause:** The validator fetches traces containing `rpc.request` from Tempo and -checks if any child span is named `rpc.process`. Both spans are emitted (they pass -individual checks), but the parent-child relationship isn't established. - -**Investigation:** - -- Check `src/xrpld/rpc/detail/ServerHandler.cpp` — `rpc.request` (line 271) and - `rpc.process` (line 573) are in the same file. Verify that `rpc.process` is created - as a child of `rpc.request` (i.e., its parent context is set). -- The issue may be that `rpc.process` creates a new root span instead of linking to the - `rpc.request` span context. - -**Files to modify:** - -- Possibly `src/xrpld/rpc/detail/ServerHandler.cpp` — fix span parenting -- OR `docker/telemetry/workload/validate_telemetry.py` — if hierarchy check logic is wrong - ---- - -## Category 4: Dashboard 404s (5 failures) - -**Symptoms:** - -``` -[FAIL] dashboard.xrpld-statsd-node-health: HTTP 404 -[FAIL] dashboard.xrpld-statsd-network: HTTP 404 -[FAIL] dashboard.xrpld-statsd-rpc: HTTP 404 -[FAIL] dashboard.xrpld-statsd-overlay-detail: HTTP 404 -[FAIL] dashboard.xrpld-statsd-ledger-sync: HTTP 404 -``` - -**Root Cause:** Dashboard UIDs were renamed from `xrpld-statsd-*` to `xrpld-system-*` -but `expected_metrics.json` still references the old names. - -**Actual UIDs in `docker/telemetry/grafana/dashboards/`:** - -| Expected (in expected_metrics.json) | Actual (in dashboard JSON) | -| ----------------------------------- | ----------------------------- | -| `xrpld-statsd-node-health` | `xrpld-system-node-health` | -| `xrpld-statsd-network` | `xrpld-system-network` | -| `xrpld-statsd-rpc` | `xrpld-system-rpc` | -| `xrpld-statsd-overlay-detail` | `xrpld-system-overlay-detail` | -| `xrpld-statsd-ledger-sync` | `xrpld-system-ledger-sync` | - -**Fix:** Update the 5 UIDs in `expected_metrics.json` → `grafana_dashboards.uids[]`. - -**Files to modify:** - -- `docker/telemetry/workload/expected_metrics.json` — update dashboard UIDs - ---- - -## Execution Order - -1. **Category 4 (Dashboard UIDs)** — trivial rename, no investigation needed -2. **Category 1 (StatsD/OTLP metrics)** — requires investigation to choose Option A vs B - and capture actual metric names -3. **Category 2 (Missing tx spans)** — requires investigation into transaction code paths -4. **Category 3 (Span hierarchy)** — requires investigation into span context propagation - -## Branch - -All changes go on: `pratik/otel-phase10-workload-validation` -Worktree: `/tmp/otel-phase10-iter`