THE GREAT CULLING: Remove all OverlayFS and delta caching logic.
After extensive investigation and testing, we determined that OverlayFS
file-level layering is fundamentally incompatible with ccache's access
patterns:
- ccache opens files with O_RDWR → kernel must provide writable file handle
- OverlayFS must copy files to upper layer immediately (can't wait)
- Even with metacopy=on, metadata-only files still appear in upper layer
- Result: ~366MB deltas instead of tiny incremental diffs
The fundamental constraint: cannot have all three of:
1. Read-only lower layer (for base sharing)
2. Writable file handles (for O_RDWR)
3. Minimal deltas (for efficient caching)
Changes:
- Removed all OverlayFS mounting/unmounting logic
- Removed workspace and registry tracking
- Removed delta creation and restoration
- Removed use-deltas parameter
- Simplified to direct tar/extract workflow
Before: 726 lines across cache actions
After: 321 lines (-55% reduction)
Benefits:
- ✅ Simpler architecture (direct tar/extract)
- ✅ More maintainable (less code, less complexity)
- ✅ More reliable (fewer moving parts)
- ✅ Same performance (base-only was already used)
- ✅ Clear path forward (restic/borg for future optimization)
Current state works great:
- Build times: 20-30 min → 2-5 min (80% improvement)
- Cache sizes: ~323-609 MB per branch (with zst compression)
- S3 costs: acceptable for current volume
If bandwidth costs become problematic, migrate to restic/borg for
chunk-level deduplication (completely different architecture).
OverlayFS operates at FILE LEVEL:
- File modified by 8 bytes → Copy entire 500 KB file to delta
- ccache updates LRU metadata (8 bytes) on every cache hit
- 667 cache hits = 667 files copied = 324 MB delta (nearly full cache)
What we need is BYTE LEVEL deltas:
- File modified by 8 bytes → Delta is 8 bytes
- 100% cache hit scenario: 10 KB delta (not 324 MB)
OverlayFS is the wrong abstraction for ccache. Binary diff tools (rsync,
xdelta3, borg) would work better, but require significant rework.
For now: Disable deltas, use base-only caching
- Base cache restore still provides massive speed boost (2-5 min vs 20-30)
- Simpler, more reliable
- Saves S3 bandwidth (no 324 MB delta uploads)
Future: Research binary diff strategies (see Gemini prompt in session notes)
PROBLEM: Profile was created before cache restore, then immediately
overwritten by cached profile. This meant we were using a potentially
stale cached profile instead of fresh configuration.
SOLUTION: Move profile creation into dependencies action after restore
- Dependencies action now takes compiler params as inputs
- Profile created AFTER cache restore (overwrites any cached profile)
- Ensures fresh profile with correct compiler settings for each job
CHANGES:
- Dependencies action: Add inputs (os, arch, compiler, compiler_version, cc, cxx)
- Dependencies action: Add 'Configure Conan' step after cache restore
- Dependencies action: Support both Linux and macOS profile generation
- Nix workflow: Remove 'Configure Conan' step, pass compiler params
- macOS workflow: Detect compiler version, pass to dependencies action
Benefits:
✅ Profile always fresh and correct for current matrix config
✅ Cache still includes .conan.db and other important files
✅ Self-contained dependencies action (easier to understand)
✅ Works for both Linux (with explicit cc/cxx) and macOS (auto-detect)
The check-conanfile-changes logic prevented cache saves on feature
branches unless conanfile changed, but this was broken because:
1. Cache keys don't include branch name (shared across branches)
2. S3 immutability makes this unnecessary (first-write-wins)
3. It prevented bootstrap saves on feature branches
Also removed unused safe-branch step (dead code from copy-paste)
- Updated xahau-ga-dependencies to use xahau-actions-cache-restore/save
- Updated xahau-ga-build to use custom cache for ccache (4 operations)
- Added AWS credential inputs to both actions
- Main workflow now passes AWS secrets to cache actions
- Removed legacy ~/.conan path (Conan 2 uses ~/.conan2 only)
- All cache keys remain unchanged for compatibility
