ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): address the PR review findings

Browse Source 
Four defects from the automated review on PR #7875, each verified against the
current tree before fixing (one further comment, the row-63 dashboard overlap,
was already fixed by an earlier commit and needed nothing).

1. Rule J could not detect an instrument-kind mismatch. instrument_kinds() wrote
   `kinds[wire] = ...`, so a wire name created through two different factories
   kept only the kind visited last and whichever emit site the file walk reached
   last silently decided the verdict. It now collects a set per name and reports
   the conflict itself -- one name exporting two instruments is the defect, and
   no suffix can be correct for both. Added a regression test that builds a name
   as both a counter and an observable gauge and asserts the message names both.

2. A duplicate connection was reported as `tls_fail`. The TLS handshake had in
   fact succeeded; PeerFinder simply already held a slot for that address, which
   is ordinary churn on a healthy node. Conflating the two made a rising
   `tls_fail` unreadable -- it could mean unreachable peers or merely a busy
   PeerFinder, and those need opposite responses. Added a distinct `duplicate`
   outcome and carried the widened vocabulary through every place that
   enumerates it: the panel description, both filter descriptions, the runbook
   branch table, the runbook outcome list and the expected_spans note. The
   `dial_outcome` template variable is a label_values() query, so it picks the
   new value up on its own.

3. ConnectAttempt::onShutdown had no `operation_aborted` guard, unlike the five
   other handlers in the same file. A clean teardown was therefore counted as
   `upgrade_fail`, inflating that outcome on any node shutting down with dials in
   flight.

4. ValidatorSite used the raw configured URI as a Prometheus label.
   [validator_list_sites] accepts credentials in the URI and ParsedUrl keeps them
   in username/password, so a configured `https://user:pass@host` would have
   copied the secret into a metric label and on into the collector, Prometheus
   and every dashboard. The label is now rebuilt from scheme, host, port and
   path -- everything needed to tell one site apart, and nothing more.

Verified: naming checker exits 0 with Rule J still passing all 40 real
instrument names; its unit tests now number 139 and all pass; 15 dashboards
validate; both workload JSON files parse; clang-tidy over the full compile
database reports no finding on either changed .cpp; pre-commit passes.

Not verified: not compiled. Item 4 introduces string concatenation and item 2 a
new constexpr, so CI's build is the first real check on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-28 17:04:28 +01:00
parent 0ddb4e2686
commit f649670ef7
8 changed files with 96 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docker/telemetry/grafana/dashboards/ledger-sync-health.json
View File

@@ -230,7 +230,7 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Count of outbound peer connection attempts, split by terminal outcome.*\n\n###### How it's computed:\n*Count over the selected range of finished dials grouped by outcome, per node. Filter the outcome set with the Dial Outcome variable.*\n\n###### Reading it:\n*Connected should dominate. The failure lines name the stage that broke: tcp_fail (no route or refused), tls_fail (TLS handshake), upgrade_fail (HTTP upgrade or protocol negotiation), timeout (no terminal state in time).*\n\n###### Healthy range:\n*Connected rising to the configured peer count, then flat with failures near zero.*\n\n###### Watch for:\n*All attempts landing on one failure outcome and no connected line — the node has no outbound peers and can never sync.*\n\n###### Keywords:\n- **Outbound dial latency** *(per node)* — an outbound peer connection attempt from TCP connect through TLS to protocol upgrade; each attempt ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)",
"description": "###### What this is:\n*Count of outbound peer connection attempts, split by terminal outcome.*\n\n###### How it's computed:\n*Count over the selected range of finished dials grouped by outcome, per node. Filter the outcome set with the Dial Outcome variable.*\n\n###### Reading it:\n*Connected should dominate. The failure lines name the stage that broke: tcp_fail (no route or refused), tls_fail (TLS handshake or shared-value exchange), duplicate (already connected to that address, not a fault), upgrade_fail (HTTP upgrade or protocol negotiation), timeout (no terminal state in time).*\n\n###### Healthy range:\n*Connected rising to the configured peer count, then flat with failures near zero.*\n\n###### Watch for:\n*All attempts landing on one failure outcome and no connected line — the node has no outbound peers and can never sync.*\n\n###### Keywords:\n- **Outbound dial latency** *(per node)* — an outbound peer connection attempt from TCP connect through TLS to protocol upgrade; each attempt ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)",
"fieldConfig": {
"defaults": {
"color": {
@@ -5158,7 +5158,7 @@
{
"name": "dial_outcome",
"label": "Dial Outcome",
"description": "Filter outbound dial attempts by terminal outcome [connected / tcp_fail / tls_fail / upgrade_fail / timeout]",
"description": "Filter outbound dial attempts by terminal outcome [connected / tcp_fail / tls_fail / duplicate / upgrade_fail / timeout]",
"type": "query",
"query": "label_values(overlay_connect_total, outcome)",
"datasource": {
@@ -5518,7 +5518,7 @@
{
"name": "span_outcome",
"label": "Span Outcome",
"description": "Filter the span-derived sync panels by terminal outcome [complete / failed / timeout / abandoned / partial / refused / connected / tcp_fail / tls_fail / upgrade_fail]",
"description": "Filter the span-derived sync panels by terminal outcome [complete / failed / timeout / abandoned / partial / refused / connected / tcp_fail / tls_fail / duplicate / upgrade_fail]",
"type": "query",
"query": "label_values(span_calls_total, outcome)",
"datasource": {

2
docker/telemetry/workload/expected_spans.json
View File

@@ -395,7 +395,7 @@
"parent": null,
"required_attributes": ["remote_endpoint", "outcome", "duration_ms"],
"config_flag": "trace_peer",
"note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same five values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality."
"note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same six values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|duplicate|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality."
},
{
"name": "peer.proposal.receive",