ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): address the PR review findings

Browse Source 
Four defects from the automated review on PR #7875, each verified against the
current tree before fixing (one further comment, the row-63 dashboard overlap,
was already fixed by an earlier commit and needed nothing).

1. Rule J could not detect an instrument-kind mismatch. instrument_kinds() wrote
   `kinds[wire] = ...`, so a wire name created through two different factories
   kept only the kind visited last and whichever emit site the file walk reached
   last silently decided the verdict. It now collects a set per name and reports
   the conflict itself -- one name exporting two instruments is the defect, and
   no suffix can be correct for both. Added a regression test that builds a name
   as both a counter and an observable gauge and asserts the message names both.

2. A duplicate connection was reported as `tls_fail`. The TLS handshake had in
   fact succeeded; PeerFinder simply already held a slot for that address, which
   is ordinary churn on a healthy node. Conflating the two made a rising
   `tls_fail` unreadable -- it could mean unreachable peers or merely a busy
   PeerFinder, and those need opposite responses. Added a distinct `duplicate`
   outcome and carried the widened vocabulary through every place that
   enumerates it: the panel description, both filter descriptions, the runbook
   branch table, the runbook outcome list and the expected_spans note. The
   `dial_outcome` template variable is a label_values() query, so it picks the
   new value up on its own.

3. ConnectAttempt::onShutdown had no `operation_aborted` guard, unlike the five
   other handlers in the same file. A clean teardown was therefore counted as
   `upgrade_fail`, inflating that outcome on any node shutting down with dials in
   flight.

4. ValidatorSite used the raw configured URI as a Prometheus label.
   [validator_list_sites] accepts credentials in the URI and ParsedUrl keeps them
   in username/password, so a configured `https://user:pass@host` would have
   copied the secret into a metric label and on into the collector, Prometheus
   and every dashboard. The label is now rebuilt from scheme, host, port and
   path -- everything needed to tell one site apart, and nothing more.

Verified: naming checker exits 0 with Rule J still passing all 40 real
instrument names; its unit tests now number 139 and all pass; 15 dashboards
validate; both workload JSON files parse; clang-tidy over the full compile
database reports no finding on either changed .cpp; pre-commit passes.

Not verified: not compiled. Item 4 introduces string concatenation and item 2 a
new constexpr, so CI's build is the first real check on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-28 17:04:28 +01:00
parent 0ddb4e2686
commit f649670ef7
8 changed files with 96 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
docs/telemetry-runbook.md
View File

@@ -2752,14 +2752,14 @@ outranks every other symptom regardless of what the mode machine says.
Peer count flat at zero; _Mode Transitions by Edge_ shows the node never leaving
`disconnected`, or churning straight back to it.
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP |
| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it |
| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal |
| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake |
| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B |
| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all |
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP |
| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it |
| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `duplicate` = already connected, not a fault · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal |
| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake |
| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B |
| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all |
**Conclusion:** the node has no usable overlay. Nothing downstream can be
diagnosed until `connected` on _Outbound Dial Outcome Rate_ is non-zero. Detail:
@@ -2943,6 +2943,9 @@ first one that is wrong and fix it before reading further panels.
- `connected` — success; this is the line that must be non-zero.
- `tcp_fail` — no route, refused, or the peer port is closed or firewalled.
- `tls_fail` — the TLS handshake failed.
- `duplicate` — TLS succeeded but PeerFinder already holds a slot for
that address. Ordinary churn on a healthy node, not a failure; it is
reported separately so a rising `tls_fail` cannot be confused with it.
- `upgrade_fail` — TLS succeeded but the HTTP upgrade or protocol
negotiation was rejected. This is the outcome that pairs with step 3.
- `timeout` — the attempt never reached a terminal state.