Commit Graph

13840 Commits

Author SHA1 Message Date
Nicholas Dudfield
1474e808cb test: expect state 2 2025-10-30 09:43:26 +07:00
Nicholas Dudfield
457e633a81 feat(test): add workflow_dispatch inputs for state machine testing
Instead of requiring commits with message tags, allow manual workflow
triggering with inputs:
- state_assertion: Expected state for validation
- start_state: Force specific starting state
- clear_cache: Clear cache before running

Benefits:
- No need for empty commits to trigger tests
- Easy manual testing via GitHub Actions UI
- Still supports commit message tags for push events
- Workflow inputs take priority over commit tags
2025-10-30 09:40:26 +07:00
Nicholas Dudfield
7ea99caa19 fix(cache): trim whitespace in delta count + use s3api for tagging
Two fixes:
1. Restore: Trim whitespace from DELTA_COUNT to fix integer comparison
2. Save: Use 's3api put-object' instead of 's3 cp' to support --tagging

The aws s3 cp command doesn't support --tagging parameter.
Switched to s3api put-object which supports tagging directly.
Tags are needed for lifecycle policies (30-day eviction).
2025-10-30 09:29:02 +07:00
Nicholas Dudfield
3e5c15c172 fix(cache): handle empty cache gracefully in [ci-clear-cache]
When [ci-clear-cache] is used but no cache exists yet, grep returns
exit code 1 which causes script failure with set -e.

Add || echo "0" to handle case where no deltas exist to delete.
2025-10-30 09:25:21 +07:00
Nicholas Dudfield
52b4fb503c feat(cache): implement [ci-clear-cache] tag + auto-detecting state machine test
Cache Clearing Feature:
- Add [ci-clear-cache] commit message tag detection in restore action
- Deletes base + all deltas when tag present
- Implicit access via github.event.head_commit.message env var
- No workflow changes needed (action handles automatically)
- Commit message only (not PR title) - one-time action

State Machine Test Workflow:
- Auto-detects state by counting state files (state0.txt, state1.txt, etc.)
- Optional [state:N] assertions validate detected == expected
- [start-state:N] forces specific state for scenario testing
- Dual validation: local cache state AND S3 objects
- 4 validation checkpoints: S3 before, local after restore, after build, S3 after save
- Self-documenting: prints next steps after each run
- Supports [ci-clear-cache] integration

Usage:
  # Auto-advance (normal)
  git commit -m 'continue testing'

  # With assertion
  git commit -m 'test delta [state:2]'

  # Clear and restart
  git commit -m 'fresh start [ci-clear-cache]'

  # Jump to scenario
  git commit -m 'test from state 3 [start-state:3]'
2025-10-30 09:19:12 +07:00
Nicholas Dudfield
98123fa934 feat(cache): implement inline delta cleanup (keep only 1 per key)
- Add automatic cleanup after each delta upload
- Query all deltas for key, sort by LastModified
- Keep only latest (just uploaded), delete all older ones
- Matches restore logic (only uses latest delta)
- Minimal storage: 1 delta per key (~2GB) vs unbounded growth
- Simpler than keeping N: restore never needs older deltas
- Concurrency-safe (idempotent batch deletes)
- Eliminates need for separate cleanup workflow

Rationale: Since restore only ever uses the single latest delta,
keeping historical deltas adds complexity without benefit. This
matches GitHub Actions semantics (one 'latest' per key).
2025-10-29 14:51:28 +07:00
Nicholas Dudfield
ce7b1c4f1d feat: add custom S3+OverlayFS cache actions with configurable delta support
Implements drop-in replacement for actions/cache using S3 backend and OverlayFS for delta caching:

- xahau-actions-cache-restore: Downloads immutable base + optional latest delta
- xahau-actions-cache-save: Saves immutable bases (bootstrap/partial-match) or timestamped deltas (exact-match)

Key features:
- Immutable bases: One static base per key (first-write-wins, GitHub Actions semantics)
- Timestamped deltas: Always-timestamped to eliminate concurrency issues
- Configurable use-deltas parameter (default true):
  - true: For symbolic keys (branch-based) - massive bandwidth savings via incremental deltas
  - false: For content-based keys (hash-based) - base-only mode, no delta complexity
- Three cache modes: bootstrap, partial-match (restore-keys), exact-match
- OverlayFS integration: Automatic delta extraction via upperdir, whiteout file support
- S3 lifecycle ready: Bases tagged 'type=base', deltas tagged 'type=delta-archive'

Decision rule for use-deltas:
- Content-based discriminator (hashFiles, commit SHA) → use-deltas: false
- Symbolic discriminator (branch name, tag, PR) → use-deltas: true

Also disables existing workflows temporarily during development.
2025-10-29 13:07:40 +07:00
Nicholas Dudfield
e062dcae58 feat(wip): comment out unused secret encryption 2025-10-29 09:05:17 +07:00
Nicholas Dudfield
a9d284fec1 feat(wip): use new key names 2025-10-29 09:02:24 +07:00
Nicholas Dudfield
065d0c3e07 feat(wip): remove currently unused workflows 2025-10-29 08:57:21 +07:00
Nicholas Dudfield
4fda40b709 test: add S3 upload to overlayfs delta test
- Upload delta tarball to S3 bucket
- Test file: hello-world-first-test.tar.gz
- Uses new secret names: XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_KEY_ID/ACCESS_KEY
- Verifies upload with aws s3 ls
- Complete end-to-end test: OverlayFS → tarball → S3
2025-10-29 08:49:05 +07:00
Nicholas Dudfield
6014356d91 test: add encrypted secrets test to overlayfs workflow
- Generate random encryption key and store in GitHub Secrets via gh CLI
- Encrypt test message with GPG and commit to repo
- Decrypt in workflow using key from secrets and echo result
- Demonstrates encrypted secrets approach for SSH keys
2025-10-29 08:04:28 +07:00
Nicholas Dudfield
d790f97430 feat(wip): experiment overlayfs 2025-10-29 07:52:06 +07:00
tequ
9ed20a4f1c Refactor: SetCron to CronSet (#609) 2025.10.27-release+2405 2025-10-27 14:38:40 +10:00
tequ
89ffc1969b Add Previous fields to ltCron (#611) 2025-10-27 14:36:57 +10:00
tequ
79fdafe638 Support Cron in util_keylet Hook API (#612) 2025-10-27 14:35:01 +10:00
tequ
2a10013dfc Support 'cron' with ledger_entry RPC (#608) 2025-10-24 17:05:14 +10:00
tequ
6f148a8ac7 ExtendedHookState (#406) 2025-10-23 18:57:38 +10:00
tequ
96222baf5e Add hook header generators and CI verification workflow (#597) 2025-10-22 15:25:38 +10:00
Niq Dudfield
74477d2c13 added configurable NuDB block size support in xahaud (#601) 2025-10-22 14:15:12 +10:00
Alloy Networks
9378f1a0ad Update CONTRIBUTING.md (#599) 2025-10-21 14:20:10 +10:00
tequ
6fa6a96e3a Introduce StartTime in CronSet and improve next execution scheduling (#596) 2025-10-21 14:17:53 +10:00
RichardAH
b0fcd36bcd import_vl_keys logic fix (flap fix) (#588) 2025-10-18 16:27:05 +10:00
RichardAH
1ec31e79c9 Cron (on ledger cronjobs) (#590)
Co-authored-by: tequ <git@tequ.dev>
2025-10-17 18:45:16 +10:00
tequ
9c8b005406 fix: improve logging for transaction preflight failures in applyHook.cpp (#566) 2025-10-15 12:33:32 +10:00
tequ
687ccf4203 Remove unused variable enabled in MultiSign_test.cpp (#592) 2025-10-15 12:32:31 +10:00
Niq Dudfield
83f09fd8ab ci: add clang to build matrix [ci-nix-full-matrix] (#569) 2025-10-15 11:26:31 +10:00
tequ
15c7ad6f78 Fix Invalid Tx flags (#514) 2025-10-14 15:35:48 +10:00
Niq Dudfield
1f12b9ec5a feat(logs): add -DBEAST_ENHANCED_LOGGING with file:line numbers for JLOG macro (#552) 2025-10-14 10:44:03 +10:00
Niq Dudfield
ad0531ad6c chore: fix warnings (#509)
Co-authored-by: Denis Angell <dangell@transia.co>
Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2025-10-11 11:47:13 +10:00
tequ
e580f7cfc0 chore(vscode): enable format on save in settings.json (#578) 2025-10-11 11:43:50 +10:00
tequ
094f011006 Fix emit Hook API testcase name (#580) 2025-10-11 11:43:09 +10:00
Niq Dudfield
39d1c43901 build: upgrade openssl from 1.1.1u to 3.6.0 (#587)
Updates OpenSSL dependency to the latest 3.x series available on Conan Center.
2025-10-10 19:53:35 +10:00
J. Scott Branson
b3e6a902cb Update Sample Configuration Files in /cfg for Congruence with xahaud (#584) 2025-10-10 14:59:39 +11:00
Niq Dudfield
fa1b93bfd8 build: migrate to conan 2 (#585)
Migrates the build system from Conan 1 to Conan 2
2025-10-10 14:57:46 +11:00
tequ
92e3a927fc refactor KEYLET_LINE in utils_keylet (#502)
Fixes the use of high and low in variable names, as these are determined by ripple::keylet::line processing.

Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2025-10-09 21:02:14 +11:00
tequ
8f7ebf0377 Optimize github action cache (#544)
* optimize github action cache

* fix

* refactor: improve github actions cache optimization (#3)

- move ccache configuration logic to dedicated action
- rename conanfile-changed to should-save-conan-cache for clarity

---------

Co-authored-by: Niq Dudfield <ndudfield@gmail.com>
2025-09-08 15:53:40 +10:00
Niq Dudfield
46cf6785ab fix(tests): prevent buffer corruption from concurrent log writes (#565)
std::endl triggers flush() which calls sync() on the shared log buffer.
Multiple threads racing in sync() cause str()/str("") operations to
corrupt buffer state, leading to crashes and double frees.

Added mutex to serialize access to suite.log, preventing concurrent
sync() calls on the same buffer.
2025-09-08 13:57:49 +10:00
Niq Dudfield
3c4c9c87c5 Fix rwdb memory leak with online_delete and remove flatmap (#570)
Co-authored-by: Denis Angell <dangell@transia.co>
2025-08-26 14:00:58 +10:00
Niq Dudfield
7a790246fb fix: upgrade CI to GCC 13 and fix compilation issues, fixes #557 (#559) 2025-08-14 17:41:49 +10:00
Niq Dudfield
1a3d2db8ef fix(ci): export correct snappy version (#546) 2025-08-14 14:01:32 +10:00
tequ
2fc912d54d Make release build use conan deps where possible and hbb 4.0.1 (#516)
Co-authored-by: Denis Angell <dangell@transia.co>
Co-authored-by: Niq Dudfield <ndudfield@gmail.com>
2025-08-14 12:59:57 +10:00
Niq Dudfield
849d447a20 docs(freeze): canceling escrows with deep frozen assets is allowed (#540) 2025.7.9-release+1951 2025-07-09 13:48:59 +10:00
tequ
ee27049687 IOUIssuerWeakTSH (#388) 2025-07-09 13:48:26 +10:00
tequ
60dec74baf Add DeepFreeze test for URIToken (#539) 2025-07-09 12:49:47 +10:00
Denis Angell
9abea13649 Feature Clawback (#534) 2025-07-09 12:48:46 +10:00
Denis Angell
810e15319c Feature DeepFreeze (#536)
---------

Co-authored-by: tequ <git@tequ.dev>
2025-07-09 10:33:08 +10:00
Niq Dudfield
d593f3bef5 fix: provisional PreviousTxn{Id,LedgerSeq} double threading (#515)
---------

Co-authored-by: tequ <git@tequ.dev>
2025-07-08 18:04:39 +10:00
Niq Dudfield
1233694b6c chore: add suspicious_patterns to .scripts/pre-hook and not-suspicious filter (#525)
* chore: add suspicious_patterns to .scripts/pre-hook and not-suspicious filter

* rm: kill annoying checkpatterns job

* chore: cleanup

---------

Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2025-07-01 20:58:06 +10:00
tequ
a1d42b7380 Improve unittests (#494)
* Match unit tests on start of test name (#4634)

* For example, without this change, to run the TxQ tests, must specify
  `--unittest=TxQ1,TxQ2` on the command line. With this change, can use
  `--unittest=TxQ`, and both will be run.
* An exact match will prevent any further partial matching.
* This could have some side effects for different tests with a common
  name beginning. For example, NFToken, NFTokenBurn, NFTokenDir. This
  might be useful. If not, the shorter-named test(s) can be renamed. For
  example, NFToken to NFTokens.
* Split the NFToken, NFTokenBurn, and Offer test classes. Potentially speeds
  up parallel tests by a factor of 5.

* SetHook_test, SetHookTSH_test, XahauGenesis_test

---------

Co-authored-by: Ed Hennis <ed@ripple.com>
2025-06-30 10:03:02 +10:00