ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 14:20:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): correct the dial-outcome diagnosis and harden the site label

Browse Source 
Adversarial validation of the previous commit found one of its two code fixes
was diagnosed wrongly and the other incomplete. Both are corrected here, along
with the layers the first pass missed.

1. The new dial outcome was named for the wrong condition. It was added as
   `duplicate` on the belief that PeerFinder had already granted a slot for the
   address. It has not: `Logic::onConnected` contains exactly ONE false-returning
   path and it is the self-connect check, which logs "Logic dropping as self
   connect" (include/xrpl/peerfinder/detail/Logic.h). The duplicate check lives
   in `newOutboundSlot`, evaluated before a ConnectAttempt exists, so a real
   duplicate can never reach this branch.

   That mattered beyond the name: the previous commit told operators the outcome
   was benign churn to ignore, when it actually reports a local misconfiguration
   -- this node has its own address in [ips_fixed] or behind its advertised
   endpoint, and every dial to it is wasted. Renamed to `self_connection`,
   reusing the slug `handshake_negotiation_fail_total` already publishes for the
   same fault so it reads identically on both signals, and every description
   corrected to say so. The fail() string now reads "Self connection" too.

   The first pass also missed three enforcement and contract sites: the
   ConnectAttempt.h Doxygen state machine (which still mapped the slot branch
   onto tls_fail), the LedgerSpanNames unit test (which pinned exactly five
   values over a std::array<..., 5> and so left the new member untested), and the
   span-derived twin panel plus two reference docs that still published the old
   five-value domain.

2. The credential-free site label was incomplete twice over.
   - It appended the port, and `Resource::Resource` DEFAULTS that to 443/https
     and 80/http when the config omits one. The label would have become
     `https://vl.ripple.com:443/` where Grafana Cloud currently holds
     `https://vl.ripple.com`, silently renaming the series for every deployment
     already scraping this metric. Verified against live label values before and
     after; the port is now omitted.
   - parseUrl's path group is `(/.*)?`, greedy to end of string, so a query or
     fragment lands inside `path`. A list URL authenticated by `?token=...` would
     have leaked exactly as userinfo did. The path is now truncated at the first
     '?' or '#'.
   Also updated the MetricNames.h usage example, which still taught the raw-URI
   pattern to the next author, and the 09-doc row that described the label as the
   configured URI.

3. Rule J hardening from the same review: `classify_instrument_kind` returns an
   `other` sentinel for a non-factory macro, and storing it in the kind set could
   render a future conflict as "created as counter and other". The sentinel is
   now skipped, keeping it doing what it already did -- matching no shape rule.
   Added a second regression test whose input the pre-fix code reported as CLEAN
   (gauge-then-histogram on a `_us` name), so the guard is proven by a 0-vs-1
   difference and not only by a changed message. Both new tests were run against
   a reconstructed last-wins implementation and both fail against it.
   Documented the conflict class in the Rule J rows of the checker README and
   CONTRIBUTING, which previously described only the suffix conventions.

Verified: naming checker exits 0 with Rule J passing all 40 real names; 140
checker tests pass; 15 dashboards validate; both workload JSON files parse;
clang-tidy over the full compile database reports no finding on any changed line
of ConnectAttempt.cpp or ValidatorSite.cpp; pre-commit passes.

Not verified: not compiled. The label change adds string truncation and the
outcome rename touches a constexpr used across three translation units, so CI's
build remains the first real check on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-28 17:40:52 +01:00
parent f649670ef7
commit b9497d05da
14 changed files with 181 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
docs/telemetry-runbook.md
View File

@@ -2752,14 +2752,14 @@ outranks every other symptom regardless of what the mode machine says.
Peer count flat at zero; _Mode Transitions by Edge_ shows the node never leaving
`disconnected`, or churning straight back to it.
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP |
| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it |
| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `duplicate` = already connected, not a fault · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal |
| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake |
| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B |
| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all |
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP |
| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it |
| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `self_connection` = we dialled our own address · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal |
| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake |
| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B |
| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all |
**Conclusion:** the node has no usable overlay. Nothing downstream can be
diagnosed until `connected` on _Outbound Dial Outcome Rate_ is non-zero. Detail:
@@ -2943,9 +2943,11 @@ first one that is wrong and fix it before reading further panels.
- `connected` — success; this is the line that must be non-zero.
- `tcp_fail` — no route, refused, or the peer port is closed or firewalled.
- `tls_fail` — the TLS handshake failed.
- `duplicate` — TLS succeeded but PeerFinder already holds a slot for
that address. Ordinary churn on a healthy node, not a failure; it is
reported separately so a rising `tls_fail` cannot be confused with it.
- `self_connection` — TLS succeeded and PeerFinder then recognised the
remote address as one of this node's own, so it had dialled itself. A
local misconfiguration (own address in `[ips_fixed]`, or behind the
advertised endpoint), not an unreachable peer; reported separately so a
rising `tls_fail` is not confused with it.
- `upgrade_fail` — TLS succeeded but the HTTP upgrade or protocol
negotiation was rejected. This is the outcome that pairs with step 3.
- `timeout` — the attempt never reached a terminal state.
@@ -3527,7 +3529,7 @@ panel it reads.
- **All series flat at zero** — normal. It means this node already held
every proposed set locally and never had to fetch one.
- **Which peer is the dial failing against?** Panel _Outbound Dial
Outcomes (span-derived, per attempt)_ (`peer.dial`). The same five
Outcomes (span-derived, per attempt)_ (`peer.dial`). The same six
outcomes as `overlay_connect_total` in Bootstrap step 2, set from the
same code path so the two cannot disagree — read the counter first for
the rate, then come here for the **identity**. The span carries