ci(telemetry): make a cluster bring-up failure diagnosable

A validation run timed out at Step 3 with only 4 of 5 nodes proposing, and
the reason was unrecoverable afterwards. Two gaps caused that.

The node-log artifact collected `node*/debug.log` but not `node*/stdout.log`.
A node that dies before its log sink opens never writes a debug.log at all,
so stdout is the only place its reason survives — and that file is written by
the harness and read by nothing, so it went to the runner and was discarded.
The failing node's log was simply absent from the artifact.

The readiness loop also fetched each node's `server_state` and threw it away,
reporting only a count. "4/5 nodes proposing" says a node is missing but not
which one, so there is nothing to grep for even once the logs are kept. The
timeout now names each node that is not proposing along with the state it
last reported, distinguishing a node that answered with a non-proposing
state from one whose RPC port did not answer at all.

Neither change affects a healthy run: the accumulator resets each attempt and
stays empty while every node is proposing.
This commit is contained in:
Pratik Mankawde
2026-08-14 22:43:03 +01:00
39 changed files with 659 additions and 363 deletions

View File

@@ -254,6 +254,11 @@ jobs:
# and `if: failure()` never fires -- which silently skipped these logs on
# every failed run, and they are the only record of why a node did not
# reach consensus.
#
# stdout.log matters as much as debug.log: a node that dies before its
# log sink opens writes no debug.log at all, so stdout is the only place
# its reason survives. A run that timed out at 4/5 nodes was left
# undiagnosable because that file was not collected.
- name: Upload node logs
if: always() && steps.validation.outcome != 'success'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -261,6 +266,7 @@ jobs:
name: xrpld-node-logs
path: |
/tmp/xrpld-validation/node*/debug.log
/tmp/xrpld-validation/node*/stdout.log
/tmp/xrpld-validation/*.log
retention-days: 7
if-no-files-found: warn