Commit Graph

73 Commits

Author SHA1 Message Date
Nicholas Dudfield
b24e4647ba fix: configure ccache after cache restore to prevent stale config
Same ordering bug as Conan profile (Session 8) - ccache config was being
created in workflow BEFORE cache restore, causing cached ccache.conf to
overwrite fresh configuration.

Changes:
- Build action: Add ccache config inputs (max_size, hash_dir, compiler_check)
- Build action: Configure ccache AFTER cache restore (overwrites cached config)
- Build action: Add "Show ccache config before build" step (debugging aid)
- Build action: Remove debug steps (past debugging phase)
- Build action: Remove ninja -v flag (past debugging phase)
- Nix workflow: Remove "Configure ccache" step (now handled in build action)
- macOS workflow: Remove "Configure ccache" step (now handled in build action)
- macOS workflow: Add missing stdlib and AWS credentials to build step
- Delete unused xahau-configure-ccache action (logic moved to build action)

Flow now matches Conan pattern:
1. Restore cache (includes potentially stale config)
2. Configure ccache (overwrites with fresh config: 2G max, hash_dir=true, compiler_check=content)
3. Show config (verification)
4. Build

This ensures fresh ccache configuration for each job, preventing issues
from cached config files with different settings.
2025-10-31 11:03:12 +07:00
Nicholas Dudfield
ef40a7f351 refactor: move Conan profile creation after cache restore
PROBLEM: Profile was created before cache restore, then immediately
overwritten by cached profile. This meant we were using a potentially
stale cached profile instead of fresh configuration.

SOLUTION: Move profile creation into dependencies action after restore
- Dependencies action now takes compiler params as inputs
- Profile created AFTER cache restore (overwrites any cached profile)
- Ensures fresh profile with correct compiler settings for each job

CHANGES:
- Dependencies action: Add inputs (os, arch, compiler, compiler_version, cc, cxx)
- Dependencies action: Add 'Configure Conan' step after cache restore
- Dependencies action: Support both Linux and macOS profile generation
- Nix workflow: Remove 'Configure Conan' step, pass compiler params
- macOS workflow: Detect compiler version, pass to dependencies action

Benefits:
 Profile always fresh and correct for current matrix config
 Cache still includes .conan.db and other important files
 Self-contained dependencies action (easier to understand)
 Works for both Linux (with explicit cc/cxx) and macOS (auto-detect)
2025-10-30 16:41:43 +07:00
Nicholas Dudfield
c46ede7c8f chore: bump CACHE_VERSION to 3 for fresh bootstrap
Triggering clean rebuild after S3 cache cleared and ccache simplified to single directory.
2025-10-30 11:57:02 +07:00
Nicholas Dudfield
0e2bc365ea refactor: simplify ccache to single directory with restore-keys fallback
Removed split ccache configuration (~/.ccache-main + ~/.ccache-current).

The split had an edge case: building a feature branch before main branch
cache is primed results in double ccache disk usage:
- Feature branch builds first → populates ~/.ccache-current (2G max_size)
- Main branch builds later → populates ~/.ccache-main (2G max_size)
- Next feature branch build → restores both caches (4G total)

This doubles ccache disk usage unnecessarily on GitHub-hosted runners.

New single-directory approach:
- Single ~/.ccache directory for all branches
- Branch-specific keys with restore-keys fallback to main branch
- Cache actions handle cross-branch sharing via partial-match mode
- Simpler and prevents double disk usage edge case

Changes:
- xahau-configure-ccache: Single cache_dir input, removed branch logic
- xahau-ga-build: Single restore/save step, uses restore-keys for fallback
- xahau-ga-nix: Removed is_main_branch parameter

Disk usage: 2G max (vs 4G worst case with split)
2025-10-30 11:24:14 +07:00
Nicholas Dudfield
631650f7eb feat(wip): add nd-experiment-overlayfs-2025-10-29 to nix push 2025-10-30 09:56:33 +07:00
Nicholas Dudfield
0b31d8e534 feat: replace actions/cache with custom S3+OverlayFS cache
- Updated xahau-ga-dependencies to use xahau-actions-cache-restore/save
- Updated xahau-ga-build to use custom cache for ccache (4 operations)
- Added AWS credential inputs to both actions
- Main workflow now passes AWS secrets to cache actions
- Removed legacy ~/.conan path (Conan 2 uses ~/.conan2 only)
- All cache keys remain unchanged for compatibility
2025-10-30 09:55:06 +07:00
Nicholas Dudfield
ecf03f4afe test: expect state 1 [ci-clear-cache] 2025-10-30 09:50:57 +07:00
Nicholas Dudfield
b801c2837d test: expect state 3 2025-10-30 09:46:17 +07:00
Nicholas Dudfield
1474e808cb test: expect state 2 2025-10-30 09:43:26 +07:00
Nicholas Dudfield
457e633a81 feat(test): add workflow_dispatch inputs for state machine testing
Instead of requiring commits with message tags, allow manual workflow
triggering with inputs:
- state_assertion: Expected state for validation
- start_state: Force specific starting state
- clear_cache: Clear cache before running

Benefits:
- No need for empty commits to trigger tests
- Easy manual testing via GitHub Actions UI
- Still supports commit message tags for push events
- Workflow inputs take priority over commit tags
2025-10-30 09:40:26 +07:00
Nicholas Dudfield
52b4fb503c feat(cache): implement [ci-clear-cache] tag + auto-detecting state machine test
Cache Clearing Feature:
- Add [ci-clear-cache] commit message tag detection in restore action
- Deletes base + all deltas when tag present
- Implicit access via github.event.head_commit.message env var
- No workflow changes needed (action handles automatically)
- Commit message only (not PR title) - one-time action

State Machine Test Workflow:
- Auto-detects state by counting state files (state0.txt, state1.txt, etc.)
- Optional [state:N] assertions validate detected == expected
- [start-state:N] forces specific state for scenario testing
- Dual validation: local cache state AND S3 objects
- 4 validation checkpoints: S3 before, local after restore, after build, S3 after save
- Self-documenting: prints next steps after each run
- Supports [ci-clear-cache] integration

Usage:
  # Auto-advance (normal)
  git commit -m 'continue testing'

  # With assertion
  git commit -m 'test delta [state:2]'

  # Clear and restart
  git commit -m 'fresh start [ci-clear-cache]'

  # Jump to scenario
  git commit -m 'test from state 3 [start-state:3]'
2025-10-30 09:19:12 +07:00
Nicholas Dudfield
ce7b1c4f1d feat: add custom S3+OverlayFS cache actions with configurable delta support
Implements drop-in replacement for actions/cache using S3 backend and OverlayFS for delta caching:

- xahau-actions-cache-restore: Downloads immutable base + optional latest delta
- xahau-actions-cache-save: Saves immutable bases (bootstrap/partial-match) or timestamped deltas (exact-match)

Key features:
- Immutable bases: One static base per key (first-write-wins, GitHub Actions semantics)
- Timestamped deltas: Always-timestamped to eliminate concurrency issues
- Configurable use-deltas parameter (default true):
  - true: For symbolic keys (branch-based) - massive bandwidth savings via incremental deltas
  - false: For content-based keys (hash-based) - base-only mode, no delta complexity
- Three cache modes: bootstrap, partial-match (restore-keys), exact-match
- OverlayFS integration: Automatic delta extraction via upperdir, whiteout file support
- S3 lifecycle ready: Bases tagged 'type=base', deltas tagged 'type=delta-archive'

Decision rule for use-deltas:
- Content-based discriminator (hashFiles, commit SHA) → use-deltas: false
- Symbolic discriminator (branch name, tag, PR) → use-deltas: true

Also disables existing workflows temporarily during development.
2025-10-29 13:07:40 +07:00
Nicholas Dudfield
e062dcae58 feat(wip): comment out unused secret encryption 2025-10-29 09:05:17 +07:00
Nicholas Dudfield
a9d284fec1 feat(wip): use new key names 2025-10-29 09:02:24 +07:00
Nicholas Dudfield
065d0c3e07 feat(wip): remove currently unused workflows 2025-10-29 08:57:21 +07:00
Nicholas Dudfield
4fda40b709 test: add S3 upload to overlayfs delta test
- Upload delta tarball to S3 bucket
- Test file: hello-world-first-test.tar.gz
- Uses new secret names: XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_KEY_ID/ACCESS_KEY
- Verifies upload with aws s3 ls
- Complete end-to-end test: OverlayFS → tarball → S3
2025-10-29 08:49:05 +07:00
Nicholas Dudfield
6014356d91 test: add encrypted secrets test to overlayfs workflow
- Generate random encryption key and store in GitHub Secrets via gh CLI
- Encrypt test message with GPG and commit to repo
- Decrypt in workflow using key from secrets and echo result
- Demonstrates encrypted secrets approach for SSH keys
2025-10-29 08:04:28 +07:00
Nicholas Dudfield
d790f97430 feat(wip): experiment overlayfs 2025-10-29 07:52:06 +07:00
tequ
96222baf5e Add hook header generators and CI verification workflow (#597) 2025-10-22 15:25:38 +10:00
Niq Dudfield
83f09fd8ab ci: add clang to build matrix [ci-nix-full-matrix] (#569) 2025-10-15 11:26:31 +10:00
Niq Dudfield
fa1b93bfd8 build: migrate to conan 2 (#585)
Migrates the build system from Conan 1 to Conan 2
2025-10-10 14:57:46 +11:00
tequ
8f7ebf0377 Optimize github action cache (#544)
* optimize github action cache

* fix

* refactor: improve github actions cache optimization (#3)

- move ccache configuration logic to dedicated action
- rename conanfile-changed to should-save-conan-cache for clarity

---------

Co-authored-by: Niq Dudfield <ndudfield@gmail.com>
2025-09-08 15:53:40 +10:00
Niq Dudfield
7a790246fb fix: upgrade CI to GCC 13 and fix compilation issues, fixes #557 (#559) 2025-08-14 17:41:49 +10:00
tequ
2fc912d54d Make release build use conan deps where possible and hbb 4.0.1 (#516)
Co-authored-by: Denis Angell <dangell@transia.co>
Co-authored-by: Niq Dudfield <ndudfield@gmail.com>
2025-08-14 12:59:57 +10:00
Niq Dudfield
1233694b6c chore: add suspicious_patterns to .scripts/pre-hook and not-suspicious filter (#525)
* chore: add suspicious_patterns to .scripts/pre-hook and not-suspicious filter

* rm: kill annoying checkpatterns job

* chore: cleanup

---------

Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2025-07-01 20:58:06 +10:00
Denis Angell
a5ea86fdfc Add Conan Building For Development (#432) 2025-05-14 14:00:20 +10:00
tequ
989532702d Update clang-format workflow (#490) 2025-04-17 16:16:59 +10:00
Niq Dudfield
ef77b02d7f CI Release Builder (#455) 2025-03-11 13:19:28 +01:00
Wietse Wind
1fb1a99ea2 Update build-in-docker.yml 2025-02-05 08:23:49 +01:00
Wietse Wind
fa71bda29c Artifact v4 continue on error 2025-02-01 08:58:13 +01:00
Wietse Wind
412593d7bc Update artifact 2025-02-01 08:57:48 +01:00
Wietse Wind
12d8342c34 Update artifact 2025-02-01 08:57:25 +01:00
Denis Angell
79d83bd424 fix240911 (#363) 2024-09-11 13:43:03 +10:00
Wietse Wind
849a4435e0 CI Split jobs with prev job dependency & CI on jshooks (#320)
* CI on `jshooks` branch

* CI Split jobs with prev job dependency

* No multi branch worker in parallel

---------

Co-authored-by: Denis Angell <dangell@transia.co>
2024-05-29 13:45:59 +02:00
Wietse Wind
247e9d98bf CI on jshooks branch (#317) 2024-05-24 10:10:40 +10:00
Wietse Wind
caa8b382d8 🤦 2024-02-22 23:28:19 +01:00
Wietse Wind
82e04073be Revert checkout v3 2024-02-14 15:21:43 +01:00
Wietse Wind
e1b78f9682 Do clean 2024-02-14 15:20:24 +01:00
Wietse Wind
901d1d4e8d Update checkout CI to v4 2024-02-14 15:17:45 +01:00
Wietse Wind
66ee96d456 Build on release after all 2023-12-29 15:43:33 +01:00
Wietse Wind
b476aea55b Do not auto build on release 2023-12-29 15:39:48 +01:00
Denis Angell
b77b0e70e3 Add Script to check for suspicious patterns (#199)
* Create check_keys.sh

* add to workflow
2023-11-16 16:23:55 +01:00
Denis Angell
559b504c7d Update build-in-docker.yml (#196) 2023-11-09 19:44:28 +01:00
Denis Angell
43cb255337 Update Workflow (#193) 2023-11-09 19:01:50 +01:00
Denis Angell
2a18ec563d Fix HBB workflow (#191)
* Update build-in-docker.yml

* Update build-in-docker.yml

* Update build-in-docker.yml

* Update build-core.sh

* update workflow

* Update build-core.sh

* Update build-core.sh

* Update build-core.sh

* Update build-core.sh

* update workflow

* fix workflow

* Update xahaud.binary.dockerfile

* fixup

* fixup

* fixup

* fixup

* fixup

* fixup

* Update build-in-docker.yml
2023-11-09 12:20:51 +01:00
Denis Angell
04ceb5af9a update workflow (#175) 2023-11-01 17:04:13 +01:00
Denis Angell
70bd7c2ce7 Reintroduce Clang-Format & Levelization (#171)
* clang-format

* levelization

* clang-format

* update workflow (#172)

* update workflow

* Update build-in-docker.yml

* fix from `clang-format`

* Update Enum.h
2023-11-01 14:12:24 +01:00
Wietse Wind
dc29234ff5 Disable publisher: http://build.xahau.tech/ 2023-04-12 15:28:57 +02:00
Richard Holland
311d0b8bf1 Revert "Use the Conan package manager (#4367)"
This reverts commit c3a9f3dbf3.
2023-04-06 09:46:00 +00:00
Richard Holland
a5ca117ff6 Merge remote-tracking branch 'ripple/develop' into dev 2023-04-06 09:36:24 +00:00