Compare commits

...

7 Commits

Author SHA1 Message Date
Nicholas Dudfield
83f6bc64e1 fix: restore ninja -v flag for compile command visibility
Removed in b24e4647b under naive assumption we were "past debugging phase".

Reality: debugging is never done, verbose output is invaluable, costs nothing.
Keep -v flag permanently.
2025-10-31 13:09:53 +07:00
Nicholas Dudfield
be6fad9692 fix: revert to PID-based temp files (was working before)
Reverts the unnecessary mktemp change from 638cb0afe that broke cache saving.

What happened:
- Original delta code used $$ (PID) for temp files: DELTA_TARBALL="/tmp/...-$$.tar.zst"
- This creates a STRING, not a file - zstd creates the file when writing
- When removing deltas (638cb0afe), I unnecessarily changed to mktemp for "better practice"
- mktemp CREATES an empty file - zstd refuses to overwrite it
- Result: "already exists; not overwritten" error

Why it seemed to work:
- Immutability check skipped save for existing caches
- Upload code path never executed during testing
- Bug only appeared when actually trying to create new cache

The fix:
- Revert to PID-based naming ($$) that was working
- Don't fix what isn't broken

Applies to both save and restore actions for consistency.
2025-10-31 12:10:13 +07:00
Nicholas Dudfield
b24e4647ba fix: configure ccache after cache restore to prevent stale config
Same ordering bug as Conan profile (Session 8) - ccache config was being
created in workflow BEFORE cache restore, causing cached ccache.conf to
overwrite fresh configuration.

Changes:
- Build action: Add ccache config inputs (max_size, hash_dir, compiler_check)
- Build action: Configure ccache AFTER cache restore (overwrites cached config)
- Build action: Add "Show ccache config before build" step (debugging aid)
- Build action: Remove debug steps (past debugging phase)
- Build action: Remove ninja -v flag (past debugging phase)
- Nix workflow: Remove "Configure ccache" step (now handled in build action)
- macOS workflow: Remove "Configure ccache" step (now handled in build action)
- macOS workflow: Add missing stdlib and AWS credentials to build step
- Delete unused xahau-configure-ccache action (logic moved to build action)

Flow now matches Conan pattern:
1. Restore cache (includes potentially stale config)
2. Configure ccache (overwrites with fresh config: 2G max, hash_dir=true, compiler_check=content)
3. Show config (verification)
4. Build

This ensures fresh ccache configuration for each job, preventing issues
from cached config files with different settings.
2025-10-31 11:03:12 +07:00
Nicholas Dudfield
638cb0afe5 refactor: remove OverlayFS delta caching entirely
THE GREAT CULLING: Remove all OverlayFS and delta caching logic.

After extensive investigation and testing, we determined that OverlayFS
file-level layering is fundamentally incompatible with ccache's access
patterns:

- ccache opens files with O_RDWR → kernel must provide writable file handle
- OverlayFS must copy files to upper layer immediately (can't wait)
- Even with metacopy=on, metadata-only files still appear in upper layer
- Result: ~366MB deltas instead of tiny incremental diffs

The fundamental constraint: cannot have all three of:
1. Read-only lower layer (for base sharing)
2. Writable file handles (for O_RDWR)
3. Minimal deltas (for efficient caching)

Changes:
- Removed all OverlayFS mounting/unmounting logic
- Removed workspace and registry tracking
- Removed delta creation and restoration
- Removed use-deltas parameter
- Simplified to direct tar/extract workflow

Before: 726 lines across cache actions
After:  321 lines (-55% reduction)

Benefits:
-  Simpler architecture (direct tar/extract)
-  More maintainable (less code, less complexity)
-  More reliable (fewer moving parts)
-  Same performance (base-only was already used)
-  Clear path forward (restic/borg for future optimization)

Current state works great:
- Build times: 20-30 min → 2-5 min (80% improvement)
- Cache sizes: ~323-609 MB per branch (with zst compression)
- S3 costs: acceptable for current volume

If bandwidth costs become problematic, migrate to restic/borg for
chunk-level deduplication (completely different architecture).
2025-10-31 10:30:31 +07:00
Nicholas Dudfield
bd384e6bc1 Revert "feat: enable metacopy=on to test metadata-only copy-up"
This reverts commit 4c546e5d91.
2025-10-31 09:51:27 +07:00
Nicholas Dudfield
4c546e5d91 feat: enable metacopy=on to test metadata-only copy-up
Mount OverlayFS with metacopy=on option (kernel 4.2+, supported on ubuntu-22.04).
This prevents full file copy-up when files are opened with O_RDWR but not modified.

Expected behavior:
- ccache opens cache files with write access
- OverlayFS creates metadata-only entry in upper layer
- Full copy-up only happens if data is actually written
- Should dramatically reduce delta sizes from ~324 MB to ~KB

Re-enabled use-deltas for ccache to test this optimization.
Conan remains base-only (hash-based keys mean exact match most of the time).

If successful, deltas should be tiny for cache hit scenarios.
2025-10-31 09:36:09 +07:00
Nicholas Dudfield
28727b3f86 perf: disable delta caching (use base-only mode)
OverlayFS operates at FILE LEVEL:
- File modified by 8 bytes → Copy entire 500 KB file to delta
- ccache updates LRU metadata (8 bytes) on every cache hit
- 667 cache hits = 667 files copied = 324 MB delta (nearly full cache)

What we need is BYTE LEVEL deltas:
- File modified by 8 bytes → Delta is 8 bytes
- 100% cache hit scenario: 10 KB delta (not 324 MB)

OverlayFS is the wrong abstraction for ccache. Binary diff tools (rsync,
xdelta3, borg) would work better, but require significant rework.

For now: Disable deltas, use base-only caching
- Base cache restore still provides massive speed boost (2-5 min vs 20-30)
- Simpler, more reliable
- Saves S3 bandwidth (no 324 MB delta uploads)

Future: Research binary diff strategies (see Gemini prompt in session notes)
2025-10-31 08:11:47 +07:00
7 changed files with 133 additions and 569 deletions

View File

@@ -1,5 +1,5 @@
name: 'Xahau Cache Restore (S3 + OverlayFS)' name: 'Xahau Cache Restore (S3)'
description: 'Drop-in replacement for actions/cache/restore using S3 and OverlayFS for delta caching' description: 'Drop-in replacement for actions/cache/restore using S3 storage'
inputs: inputs:
path: path:
@@ -28,10 +28,6 @@ inputs:
description: 'Check if a cache entry exists for the given input(s) without downloading it' description: 'Check if a cache entry exists for the given input(s) without downloading it'
required: false required: false
default: 'false' default: 'false'
use-deltas:
description: 'Enable delta caching (download/upload incremental changes). Set to false for base-only caching.'
required: false
default: 'true'
# Note: Composite actions can't access secrets.* directly - must be passed from workflow # Note: Composite actions can't access secrets.* directly - must be passed from workflow
aws-access-key-id: aws-access-key-id:
description: 'AWS Access Key ID for S3 access' description: 'AWS Access Key ID for S3 access'
@@ -48,13 +44,13 @@ outputs:
description: 'The key that was used to restore the cache (may be from restore-keys)' description: 'The key that was used to restore the cache (may be from restore-keys)'
value: ${{ steps.restore-cache.outputs.cache-primary-key }} value: ${{ steps.restore-cache.outputs.cache-primary-key }}
cache-matched-key: cache-matched-key:
description: 'The key that matched (same as cache-primary-key for compatibility)' description: 'The key that was used to restore the cache (exact or prefix match)'
value: ${{ steps.restore-cache.outputs.cache-primary-key }} value: ${{ steps.restore-cache.outputs.cache-matched-key }}
runs: runs:
using: 'composite' using: 'composite'
steps: steps:
- name: Restore cache from S3 with OverlayFS - name: Restore cache from S3
id: restore-cache id: restore-cache
shell: bash shell: bash
env: env:
@@ -67,133 +63,42 @@ runs:
TARGET_PATH: ${{ inputs.path }} TARGET_PATH: ${{ inputs.path }}
FAIL_ON_MISS: ${{ inputs.fail-on-cache-miss }} FAIL_ON_MISS: ${{ inputs.fail-on-cache-miss }}
LOOKUP_ONLY: ${{ inputs.lookup-only }} LOOKUP_ONLY: ${{ inputs.lookup-only }}
USE_DELTAS: ${{ inputs.use-deltas }}
COMMIT_MSG: ${{ github.event.head_commit.message }}
run: | run: |
set -euo pipefail set -euo pipefail
echo "==========================================" echo "=========================================="
echo "Xahau Cache Restore (S3 + OverlayFS)" echo "Xahau Cache Restore (S3)"
echo "==========================================" echo "=========================================="
echo "Target path: ${TARGET_PATH}" echo "Target path: ${TARGET_PATH}"
echo "Primary key: ${CACHE_KEY}" echo "Cache key: ${CACHE_KEY}"
echo "S3 bucket: s3://${S3_BUCKET}" echo "S3 bucket: s3://${S3_BUCKET}"
echo "Use deltas: ${USE_DELTAS}"
echo "" echo ""
# Normalize target path (expand tilde and resolve to absolute path) # Normalize target path (expand tilde and resolve to absolute path)
# This ensures consistent path comparison in the mount registry
if [[ "${TARGET_PATH}" == ~* ]]; then if [[ "${TARGET_PATH}" == ~* ]]; then
# Expand tilde manually (works even if directory doesn't exist yet)
TARGET_PATH="${HOME}${TARGET_PATH:1}" TARGET_PATH="${HOME}${TARGET_PATH:1}"
fi fi
TARGET_PATH=$(realpath -m "${TARGET_PATH}")
echo "Normalized target path: ${TARGET_PATH}" echo "Normalized target path: ${TARGET_PATH}"
echo ""
# Generate unique cache workspace # Function to try restoring a cache key
CACHE_HASH=$(echo "${CACHE_KEY}" | md5sum | cut -d' ' -f1)
CACHE_WORKSPACE="/tmp/xahau-cache-${CACHE_HASH}"
echo "Cache workspace: ${CACHE_WORKSPACE}"
# Check for [ci-clear-cache] tag in commit message
if echo "${COMMIT_MSG}" | grep -q '\[ci-clear-cache\]'; then
echo ""
echo "🗑️ [ci-clear-cache] detected in commit message"
echo "Clearing cache for key: ${CACHE_KEY}"
echo ""
# Delete base layer
S3_BASE_KEY="s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst"
if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
echo "Deleting base layer: ${S3_BASE_KEY}"
aws s3 rm "${S3_BASE_KEY}" --region "${S3_REGION}" 2>/dev/null || true
echo "✓ Base layer deleted"
else
echo " No base layer found to delete"
fi
# Delete all delta layers for this key
echo "Deleting all delta layers matching: ${CACHE_KEY}-delta-*"
DELTA_COUNT=$(aws s3 ls "s3://${S3_BUCKET}/" --region "${S3_REGION}" | grep "${CACHE_KEY}-delta-" | wc -l || echo "0")
DELTA_COUNT=$(echo "${DELTA_COUNT}" | tr -d ' \n') # Trim whitespace
if [ "${DELTA_COUNT}" -gt 0 ]; then
aws s3 rm "s3://${S3_BUCKET}/" --recursive \
--exclude "*" \
--include "${CACHE_KEY}-delta-*" \
--region "${S3_REGION}" 2>/dev/null || true
echo "✓ Deleted ${DELTA_COUNT} delta layer(s)"
else
echo " No delta layers found to delete"
fi
echo ""
echo "✅ Cache cleared successfully"
echo "Build will proceed from scratch (bootstrap mode)"
echo ""
fi
# Create OverlayFS directory structure
mkdir -p "${CACHE_WORKSPACE}"/{base,upper,work,merged}
# Function to try downloading from S3
try_restore_key() { try_restore_key() {
local try_key="$1" local key=$1
local s3_base="s3://${S3_BUCKET}/${try_key}-base.tar.zst" local s3_key="s3://${S3_BUCKET}/${key}-base.tar.zst"
echo "Trying cache key: ${try_key}" echo "Checking for key: ${key}"
# Check if base exists (one base per key, immutable)
echo "Checking for base layer..."
if aws s3 ls "${s3_base}" --region "${S3_REGION}" >/dev/null 2>&1; then
echo "✓ Found base layer: ${s3_base}"
if [ "${LOOKUP_ONLY}" = "true" ]; then
echo "Lookup-only mode: cache exists, skipping download"
return 0
fi
# Download base layer
echo "Downloading base layer..."
aws s3 cp "${s3_base}" /tmp/cache-base.tar.zst --region "${S3_REGION}" --quiet
# Extract base layer
echo "Extracting base layer..."
tar -xf /tmp/cache-base.tar.zst -C "${CACHE_WORKSPACE}/base"
rm /tmp/cache-base.tar.zst
# Query for latest timestamped delta (only if use-deltas enabled)
if [ "${USE_DELTAS}" = "true" ]; then
echo "Querying for latest delta..."
LATEST_DELTA=$(aws s3api list-objects-v2 \
--bucket "${S3_BUCKET}" \
--prefix "${try_key}-delta-" \
--region "${S3_REGION}" \
--query 'sort_by(Contents, &LastModified)[-1].Key' \
--output text 2>/dev/null || echo "")
if [ -n "${LATEST_DELTA}" ] && [ "${LATEST_DELTA}" != "None" ]; then
echo "✓ Found latest delta: ${LATEST_DELTA}"
echo "Downloading delta layer..."
aws s3 cp "s3://${S3_BUCKET}/${LATEST_DELTA}" /tmp/cache-delta.tar.zst --region "${S3_REGION}" --quiet
echo "Extracting delta layer..."
tar -xf /tmp/cache-delta.tar.zst -C "${CACHE_WORKSPACE}/upper" 2>/dev/null || true
rm /tmp/cache-delta.tar.zst
else
echo " No delta layer found (this is fine for first build)"
fi
else
echo " Delta caching disabled (use-deltas: false)"
fi
if aws s3 ls "${s3_key}" --region "${S3_REGION}" >/dev/null 2>&1; then
echo "✓ Found cache: ${s3_key}"
return 0 return 0
else else
echo "✗ No base layer found for key: ${try_key}" echo "✗ Not found: ${key}"
return 1 return 1
fi fi
} }
# Try primary key first # Try exact match first
MATCHED_KEY="" MATCHED_KEY=""
EXACT_MATCH="false" EXACT_MATCH="false"
@@ -208,12 +113,8 @@ runs:
echo "" echo ""
echo "Primary key not found, trying restore-keys..." echo "Primary key not found, trying restore-keys..."
# Split restore-keys by newline
while IFS= read -r restore_key; do while IFS= read -r restore_key; do
# Skip empty lines
[ -z "${restore_key}" ] && continue [ -z "${restore_key}" ] && continue
# Trim whitespace
restore_key=$(echo "${restore_key}" | xargs) restore_key=$(echo "${restore_key}" | xargs)
if try_restore_key "${restore_key}"; then if try_restore_key "${restore_key}"; then
@@ -231,7 +132,6 @@ runs:
if [ -z "${MATCHED_KEY}" ]; then if [ -z "${MATCHED_KEY}" ]; then
echo "" echo ""
echo "❌ No cache found for key: ${CACHE_KEY}" echo "❌ No cache found for key: ${CACHE_KEY}"
echo "This is BOOTSTRAP mode - first build for this cache key"
if [ "${FAIL_ON_MISS}" = "true" ]; then if [ "${FAIL_ON_MISS}" = "true" ]; then
echo "fail-on-cache-miss is enabled, failing workflow" echo "fail-on-cache-miss is enabled, failing workflow"
@@ -241,16 +141,11 @@ runs:
# Set outputs for cache miss # Set outputs for cache miss
echo "cache-hit=false" >> $GITHUB_OUTPUT echo "cache-hit=false" >> $GITHUB_OUTPUT
echo "cache-primary-key=" >> $GITHUB_OUTPUT echo "cache-primary-key=" >> $GITHUB_OUTPUT
echo "cache-matched-key=" >> $GITHUB_OUTPUT
# Create empty cache directory for bootstrap # Create empty cache directory
mkdir -p "${TARGET_PATH}" mkdir -p "${TARGET_PATH}"
# Record bootstrap mode for save action
# Format: path:workspace:matched_key:primary_key:exact_match:use_deltas
# For bootstrap: workspace="bootstrap", matched_key=primary_key, exact_match=false
MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt"
echo "${TARGET_PATH}:bootstrap:${CACHE_KEY}:${CACHE_KEY}:false:${USE_DELTAS}" >> "${MOUNT_REGISTRY}"
echo "" echo ""
echo "==========================================" echo "=========================================="
echo "Cache restore completed (bootstrap mode)" echo "Cache restore completed (bootstrap mode)"
@@ -262,36 +157,30 @@ runs:
# If lookup-only, we're done # If lookup-only, we're done
if [ "${LOOKUP_ONLY}" = "true" ]; then if [ "${LOOKUP_ONLY}" = "true" ]; then
echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
echo "cache-primary-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT echo "cache-primary-key=${CACHE_KEY}" >> $GITHUB_OUTPUT
echo "cache-matched-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
# Clean up workspace
rm -rf "${CACHE_WORKSPACE}"
echo "" echo ""
echo "==========================================" echo "=========================================="
echo "Cache lookup completed (lookup-only mode)" echo "Cache lookup completed (lookup-only mode)"
echo "Cache exists: ${MATCHED_KEY}"
echo "==========================================" echo "=========================================="
exit 0 exit 0
fi fi
# Mount OverlayFS # Download and extract cache
S3_KEY="s3://${S3_BUCKET}/${MATCHED_KEY}-base.tar.zst"
TEMP_TARBALL="/tmp/xahau-cache-restore-$$.tar.zst"
echo "" echo ""
echo "Mounting OverlayFS..." echo "Downloading cache..."
sudo mount -t overlay overlay \ aws s3 cp "${S3_KEY}" "${TEMP_TARBALL}" --region "${S3_REGION}"
-o lowerdir="${CACHE_WORKSPACE}/base",upperdir="${CACHE_WORKSPACE}/upper",workdir="${CACHE_WORKSPACE}/work" \
"${CACHE_WORKSPACE}/merged"
# Verify mount TARBALL_SIZE=$(du -h "${TEMP_TARBALL}" | cut -f1)
if mount | grep -q "${CACHE_WORKSPACE}/merged"; then echo "✓ Downloaded: ${TARBALL_SIZE}"
echo "✓ OverlayFS mounted successfully"
else
echo "❌ Failed to mount OverlayFS"
exit 1
fi
# Create target directory parent if needed # Create parent directory if needed
TARGET_PARENT=$(dirname "${TARGET_PATH}") mkdir -p "$(dirname "${TARGET_PATH}")"
mkdir -p "${TARGET_PARENT}"
# Remove existing target if it exists # Remove existing target if it exists
if [ -e "${TARGET_PATH}" ]; then if [ -e "${TARGET_PATH}" ]; then
@@ -299,30 +188,24 @@ runs:
rm -rf "${TARGET_PATH}" rm -rf "${TARGET_PATH}"
fi fi
# Symlink target path to merged view # Create target directory and extract
echo "Creating symlink: ${TARGET_PATH} -> ${CACHE_WORKSPACE}/merged" mkdir -p "${TARGET_PATH}"
ln -s "${CACHE_WORKSPACE}/merged" "${TARGET_PATH}" echo ""
echo "Extracting cache..."
zstd -d -c "${TEMP_TARBALL}" | tar -xf - -C "${TARGET_PATH}"
echo "✓ Cache extracted to: ${TARGET_PATH}"
# Save mount info for cleanup/save later # Cleanup
# Format: path:workspace:matched_key:primary_key:exact_match:use_deltas rm -f "${TEMP_TARBALL}"
# This tells save action whether to create new base (partial match) or just delta (exact match)
MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt"
echo "${TARGET_PATH}:${CACHE_WORKSPACE}:${MATCHED_KEY}:${CACHE_KEY}:${EXACT_MATCH}:${USE_DELTAS}" >> "${MOUNT_REGISTRY}"
# Set outputs # Set outputs
echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
echo "cache-primary-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT echo "cache-primary-key=${CACHE_KEY}" >> $GITHUB_OUTPUT
echo "cache-matched-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
# Show statistics
echo ""
echo "Cache statistics:"
echo " Base layer size: $(du -sh ${CACHE_WORKSPACE}/base 2>/dev/null | cut -f1 || echo '0')"
echo " Delta layer size: $(du -sh ${CACHE_WORKSPACE}/upper 2>/dev/null | cut -f1 || echo '0')"
echo " Merged view size: $(du -sh ${CACHE_WORKSPACE}/merged 2>/dev/null | cut -f1 || echo '0')"
echo "" echo ""
echo "==========================================" echo "=========================================="
echo "Cache restore completed successfully" echo "Cache restore completed successfully"
echo "Exact match: ${EXACT_MATCH}" echo "Cache hit: ${EXACT_MATCH}"
echo "Matched key: ${MATCHED_KEY}" echo "Matched key: ${MATCHED_KEY}"
echo "==========================================" echo "=========================================="

View File

@@ -1,5 +1,5 @@
name: 'Xahau Cache Save (S3 + OverlayFS)' name: 'Xahau Cache Save (S3)'
description: 'Drop-in replacement for actions/cache/save using S3 and OverlayFS for delta caching' description: 'Drop-in replacement for actions/cache/save using S3 storage'
inputs: inputs:
path: path:
@@ -16,10 +16,6 @@ inputs:
description: 'S3 region' description: 'S3 region'
required: false required: false
default: 'us-east-1' default: 'us-east-1'
use-deltas:
description: 'Enable delta caching (download/upload incremental changes). Set to false for base-only caching.'
required: false
default: 'true'
# Note: Composite actions can't access secrets.* directly - must be passed from workflow # Note: Composite actions can't access secrets.* directly - must be passed from workflow
aws-access-key-id: aws-access-key-id:
description: 'AWS Access Key ID for S3 access' description: 'AWS Access Key ID for S3 access'
@@ -31,7 +27,7 @@ inputs:
runs: runs:
using: 'composite' using: 'composite'
steps: steps:
- name: Save cache to S3 with OverlayFS delta - name: Save cache to S3
shell: bash shell: bash
env: env:
AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }} AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
@@ -40,12 +36,11 @@ runs:
S3_REGION: ${{ inputs.s3-region }} S3_REGION: ${{ inputs.s3-region }}
CACHE_KEY: ${{ inputs.key }} CACHE_KEY: ${{ inputs.key }}
TARGET_PATH: ${{ inputs.path }} TARGET_PATH: ${{ inputs.path }}
USE_DELTAS: ${{ inputs.use-deltas }}
run: | run: |
set -euo pipefail set -euo pipefail
echo "==========================================" echo "=========================================="
echo "Xahau Cache Save (S3 + OverlayFS)" echo "Xahau Cache Save (S3)"
echo "==========================================" echo "=========================================="
echo "Target path: ${TARGET_PATH}" echo "Target path: ${TARGET_PATH}"
echo "Cache key: ${CACHE_KEY}" echo "Cache key: ${CACHE_KEY}"
@@ -53,346 +48,63 @@ runs:
echo "" echo ""
# Normalize target path (expand tilde and resolve to absolute path) # Normalize target path (expand tilde and resolve to absolute path)
# This ensures consistent path comparison with the mount registry
if [[ "${TARGET_PATH}" == ~* ]]; then if [[ "${TARGET_PATH}" == ~* ]]; then
# Expand tilde manually (works even if directory doesn't exist yet)
TARGET_PATH="${HOME}${TARGET_PATH:1}" TARGET_PATH="${HOME}${TARGET_PATH:1}"
fi fi
echo "Normalized target path: ${TARGET_PATH}" echo "Normalized target path: ${TARGET_PATH}"
echo "" echo ""
# Find the cache workspace from mount registry # Check if target directory exists
MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt" if [ ! -d "${TARGET_PATH}" ]; then
echo "⚠️ Target directory does not exist: ${TARGET_PATH}"
if [ ! -f "${MOUNT_REGISTRY}" ]; then
echo "⚠️ No cache mounts found (mount registry doesn't exist)"
echo "This usually means cache restore was not called, or there was no cache to restore."
echo "Skipping cache save." echo "Skipping cache save."
exit 0 exit 0
fi fi
# Find entry for this path # Use static base name (one base per key, immutable)
# Format: path:workspace:matched_key:primary_key:exact_match:use_deltas S3_BASE_KEY="s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst"
# Bootstrap mode: path:bootstrap:key:key:false:true/false (workspace="bootstrap")
CACHE_WORKSPACE=""
MATCHED_KEY=""
PRIMARY_KEY=""
EXACT_MATCH=""
REGISTRY_USE_DELTAS=""
while IFS=: read -r mount_path mount_workspace mount_matched_key mount_primary_key mount_exact_match mount_use_deltas; do # Check if base already exists (immutability - first write wins)
if [ "${mount_path}" = "${TARGET_PATH}" ]; then if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
CACHE_WORKSPACE="${mount_workspace}" echo "⚠️ Cache already exists: ${S3_BASE_KEY}"
MATCHED_KEY="${mount_matched_key}" echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
PRIMARY_KEY="${mount_primary_key}" echo ""
EXACT_MATCH="${mount_exact_match}" echo "=========================================="
REGISTRY_USE_DELTAS="${mount_use_deltas}" echo "Cache save completed (already exists)"
break echo "=========================================="
fi
done < "${MOUNT_REGISTRY}"
if [ -z "${CACHE_WORKSPACE}" ] && [ -z "${MATCHED_KEY}" ]; then
echo "⚠️ No cache entry found for path: ${TARGET_PATH}"
echo "This usually means cache restore was not called for this path."
echo "Skipping cache save."
exit 0 exit 0
fi fi
# Determine cache mode # Create tarball
if [ "${CACHE_WORKSPACE}" = "bootstrap" ]; then BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst"
CACHE_MODE="bootstrap"
PRIMARY_KEY="${MATCHED_KEY}" # In bootstrap, matched_key field contains primary key echo "Creating cache tarball..."
echo "Cache mode: BOOTSTRAP (first build for this key)" tar -cf - -C "${TARGET_PATH}" . | zstd -3 -T0 -q -o "${BASE_TARBALL}"
echo "Primary key: ${PRIMARY_KEY}"
elif [ "${EXACT_MATCH}" = "false" ]; then BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1)
CACHE_MODE="partial-match" echo "✓ Cache tarball created: ${BASE_SIZE}"
echo "Cache mode: PARTIAL MATCH (restore-key used)"
echo "Cache workspace: ${CACHE_WORKSPACE}"
echo "Matched key from restore: ${MATCHED_KEY}"
echo "Primary key (will save new base): ${PRIMARY_KEY}"
else
CACHE_MODE="exact-match"
echo "Cache mode: EXACT MATCH (cache hit)"
echo "Cache workspace: ${CACHE_WORKSPACE}"
echo "Matched key: ${MATCHED_KEY}"
fi
echo "Use deltas: ${REGISTRY_USE_DELTAS}"
echo "" echo ""
# Handle different cache modes # Upload to S3
if [ "${CACHE_MODE}" = "bootstrap" ]; then echo "Uploading cache to S3..."
# Bootstrap: Save entire cache as base layer (no OverlayFS was used) echo " Key: ${CACHE_KEY}-base.tar.zst"
echo "Bootstrap mode: Creating initial base layer from ${TARGET_PATH}"
BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst" aws s3api put-object \
echo "Creating base tarball..." --bucket "${S3_BUCKET}" \
tar -cf - -C "${TARGET_PATH}" . | zstd -3 -T0 -q -o "${BASE_TARBALL}" --key "${CACHE_KEY}-base.tar.zst" \
--body "${BASE_TARBALL}" \
--tagging 'type=base' \
--region "${S3_REGION}" \
>/dev/null
BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1) echo "✓ Uploaded: ${S3_BASE_KEY}"
echo "✓ Base tarball created: ${BASE_SIZE}"
echo ""
# Use static base name (one base per key, immutable) # Cleanup
S3_BASE_KEY="s3://${S3_BUCKET}/${PRIMARY_KEY}-base.tar.zst" rm -f "${BASE_TARBALL}"
# Check if base already exists (immutability - first write wins)
if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
echo "⚠️ Base layer already exists: ${S3_BASE_KEY}"
echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
else
echo "Uploading base layer to S3..."
echo " Key: ${PRIMARY_KEY}-base.tar.zst"
aws s3api put-object \
--bucket "${S3_BUCKET}" \
--key "${PRIMARY_KEY}-base.tar.zst" \
--body "${BASE_TARBALL}" \
--tagging 'type=base' \
--region "${S3_REGION}" \
>/dev/null
echo "✓ Uploaded: ${S3_BASE_KEY}"
fi
# Cleanup
rm -f "${BASE_TARBALL}"
echo ""
echo "=========================================="
echo "Bootstrap cache save completed"
echo "Base size: ${BASE_SIZE}"
echo "Cache key: ${PRIMARY_KEY}"
echo "=========================================="
exit 0
elif [ "${CACHE_MODE}" = "partial-match" ]; then
# Partial match: Save merged view as new base ONLY (no delta)
# The delta is relative to the OLD base, not the NEW base we're creating
echo "Partial match mode: Saving new base layer for primary key"
echo "Note: Delta will NOT be saved (it's relative to old base)"
BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst"
echo "Creating base tarball from merged view..."
tar -cf - -C "${CACHE_WORKSPACE}/merged" . | zstd -3 -T0 -q -o "${BASE_TARBALL}"
BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1)
echo "✓ Base tarball created: ${BASE_SIZE}"
echo ""
# Use static base name (one base per key, immutable)
S3_BASE_KEY="s3://${S3_BUCKET}/${PRIMARY_KEY}-base.tar.zst"
# Check if base already exists (immutability - first write wins)
if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
echo "⚠️ Base layer already exists: ${S3_BASE_KEY}"
echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
else
echo "Uploading new base layer to S3..."
echo " Key: ${PRIMARY_KEY}-base.tar.zst"
aws s3api put-object \
--bucket "${S3_BUCKET}" \
--key "${PRIMARY_KEY}-base.tar.zst" \
--body "${BASE_TARBALL}" \
--tagging 'type=base' \
--region "${S3_REGION}" \
>/dev/null
echo "✓ Uploaded: ${S3_BASE_KEY}"
fi
# Cleanup
rm -f "${BASE_TARBALL}"
# Unmount and cleanup
echo ""
echo "Cleaning up..."
if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
sudo umount "${CACHE_WORKSPACE}/merged" || {
echo "⚠️ Warning: Failed to unmount ${CACHE_WORKSPACE}/merged"
echo "Attempting lazy unmount..."
sudo umount -l "${CACHE_WORKSPACE}/merged" || true
}
fi
rm -rf "${CACHE_WORKSPACE}"
# Remove from registry
if [ -f "${MOUNT_REGISTRY}" ]; then
grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
fi
echo "✓ Cleanup completed"
echo ""
echo "=========================================="
echo "Partial match cache save completed"
echo "New base created for: ${PRIMARY_KEY}"
echo "Base size: ${BASE_SIZE}"
if [ "${REGISTRY_USE_DELTAS}" = "true" ]; then
echo "Next exact-match build will create deltas from this base"
else
echo "Next exact-match build will reuse this base (base-only mode)"
fi
echo "=========================================="
exit 0
fi
# For exact-match ONLY: Save delta (if use-deltas enabled)
if [ "${CACHE_MODE}" = "exact-match" ]; then
# If deltas are disabled, just cleanup and exit
if [ "${REGISTRY_USE_DELTAS}" != "true" ]; then
echo " Delta caching disabled (use-deltas: false)"
echo "Base already exists for this key, nothing to save."
# Unmount and cleanup
echo ""
echo "Cleaning up..."
if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
sudo umount "${CACHE_WORKSPACE}/merged" 2>/dev/null || true
fi
rm -rf "${CACHE_WORKSPACE}"
# Remove from registry
if [ -f "${MOUNT_REGISTRY}" ]; then
grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
fi
echo ""
echo "=========================================="
echo "Cache save completed (base-only mode)"
echo "=========================================="
exit 0
fi
# Check if upper layer has any changes
if [ -z "$(ls -A ${CACHE_WORKSPACE}/upper 2>/dev/null)" ]; then
echo " No changes detected in upper layer (cache is unchanged)"
echo "Skipping delta upload to save bandwidth."
# Still unmount and cleanup
echo ""
echo "Cleaning up..."
sudo umount "${CACHE_WORKSPACE}/merged" 2>/dev/null || true
rm -rf "${CACHE_WORKSPACE}"
echo ""
echo "=========================================="
echo "Cache save completed (no changes)"
echo "=========================================="
exit 0
fi
# Show delta statistics
echo "Delta layer statistics:"
echo " Files changed: $(find ${CACHE_WORKSPACE}/upper -type f 2>/dev/null | wc -l)"
echo " Delta size: $(du -sh ${CACHE_WORKSPACE}/upper 2>/dev/null | cut -f1)"
echo ""
# Create delta tarball from upper layer
echo "Creating delta tarball..."
DELTA_TARBALL="/tmp/xahau-cache-delta-$$.tar.zst"
tar -cf - -C "${CACHE_WORKSPACE}/upper" . | zstd -3 -T0 -q -o "${DELTA_TARBALL}"
DELTA_SIZE=$(du -h "${DELTA_TARBALL}" | cut -f1)
echo "✓ Delta tarball created: ${DELTA_SIZE}"
echo ""
# Upload timestamped delta (no overwrites = zero concurrency issues)
TIMESTAMP=$(date +%Y%m%d%H%M%S)
COMMIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
# Use PRIMARY_KEY for delta (ensures deltas match their base)
S3_DELTA_TIMESTAMPED="s3://${S3_BUCKET}/${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst"
echo "Uploading timestamped delta to S3..."
echo " Key: ${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst"
# Upload with tag (deltas cleaned up inline - keep last 1)
aws s3api put-object \
--bucket "${S3_BUCKET}" \
--key "${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst" \
--body "${DELTA_TARBALL}" \
--tagging 'type=delta-archive' \
--region "${S3_REGION}" \
>/dev/null
echo "✓ Uploaded: ${S3_DELTA_TIMESTAMPED}"
# Inline cleanup: Keep only latest delta (the one we just uploaded)
echo ""
echo "Cleaning up old deltas (keeping only latest)..."
# List all deltas for this key, sorted by LastModified (oldest first)
ALL_DELTAS=$(aws s3api list-objects-v2 \
--bucket "${S3_BUCKET}" \
--prefix "${PRIMARY_KEY}-delta-" \
--region "${S3_REGION}" \
--query 'sort_by(Contents, &LastModified)[*].Key' \
--output json 2>/dev/null || echo "[]")
DELTA_COUNT=$(echo "${ALL_DELTAS}" | jq 'length' 2>/dev/null || echo "0")
if [ "${DELTA_COUNT}" -gt 1 ]; then
# Keep last 1 (newest), delete all older ones (all except last 1 = [0:-1])
OLD_DELTAS=$(echo "${ALL_DELTAS}" | jq -r '.[0:-1][]' 2>/dev/null)
if [ -n "${OLD_DELTAS}" ]; then
DELETE_COUNT=$((DELTA_COUNT - 1))
echo " Found ${DELETE_COUNT} old delta(s) to delete"
# Create delete batch request JSON
DELETE_OBJECTS=$(echo "${OLD_DELTAS}" | jq -R -s -c 'split("\n") | map(select(length > 0)) | map({Key: .}) | {Objects: ., Quiet: true}' 2>/dev/null)
if [ -n "${DELETE_OBJECTS}" ]; then
aws s3api delete-objects \
--bucket "${S3_BUCKET}" \
--delete "${DELETE_OBJECTS}" \
--region "${S3_REGION}" \
>/dev/null 2>&1
echo "✓ Deleted ${DELETE_COUNT} old delta(s)"
fi
fi
else
echo " Only ${DELTA_COUNT} delta(s) exist, no cleanup needed"
fi
# Cleanup delta tarball
rm -f "${DELTA_TARBALL}"
# Cleanup: Unmount OverlayFS and remove workspace
echo ""
echo "Cleaning up..."
if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
sudo umount "${CACHE_WORKSPACE}/merged" || {
echo "⚠️ Warning: Failed to unmount ${CACHE_WORKSPACE}/merged"
echo "Attempting lazy unmount..."
sudo umount -l "${CACHE_WORKSPACE}/merged" || true
}
fi
# Remove workspace
rm -rf "${CACHE_WORKSPACE}"
fi
# Remove from registry
if [ -f "${MOUNT_REGISTRY}" ]; then
grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
fi
echo "✓ Cleanup completed"
echo "" echo ""
echo "==========================================" echo "=========================================="
echo "Cache save completed successfully" echo "Cache save completed successfully"
echo "Mode: ${CACHE_MODE}" echo "Cache size: ${BASE_SIZE}"
echo "Cache key: ${PRIMARY_KEY}" echo "Cache key: ${CACHE_KEY}"
if [ -n "${DELTA_SIZE:-}" ]; then
echo "Delta size: ${DELTA_SIZE}"
fi
echo "==========================================" echo "=========================================="

View File

@@ -1,45 +0,0 @@
name: 'Configure ccache'
description: 'Sets up ccache with consistent configuration'
inputs:
cache_dir:
description: 'Path to ccache directory'
required: false
default: '~/.ccache'
max_size:
description: 'Maximum cache size'
required: false
default: '2G'
hash_dir:
description: 'Whether to include directory paths in hash'
required: false
default: 'true'
compiler_check:
description: 'How to check compiler for changes'
required: false
default: 'content'
runs:
using: 'composite'
steps:
- name: Configure ccache
shell: bash
run: |
# Use ccache's default cache_dir (~/.ccache) - don't override it
# This avoids tilde expansion issues when setting it explicitly
# Create cache directory using ccache's default
mkdir -p ~/.ccache
# Configure ccache settings (but NOT cache_dir - use default)
ccache --set-config=max_size=${{ inputs.max_size }}
ccache --set-config=hash_dir=${{ inputs.hash_dir }}
ccache --set-config=compiler_check=${{ inputs.compiler_check }}
# Note: Not setting CCACHE_DIR - let ccache use its default (~/.ccache)
# Print config for verification
ccache -p
# Zero statistics before the build
ccache -z

View File

@@ -47,6 +47,18 @@ inputs:
description: 'GCC version to use for Clang toolchain (e.g. 11, 13)' description: 'GCC version to use for Clang toolchain (e.g. 11, 13)'
required: false required: false
default: '' default: ''
ccache_max_size:
description: 'Maximum ccache size'
required: false
default: '2G'
ccache_hash_dir:
description: 'Whether to include directory paths in hash'
required: false
default: 'true'
ccache_compiler_check:
description: 'How to check compiler for changes'
required: false
default: 'content'
aws-access-key-id: aws-access-key-id:
description: 'AWS Access Key ID for S3 cache storage' description: 'AWS Access Key ID for S3 cache storage'
required: true required: true
@@ -79,6 +91,31 @@ runs:
aws-access-key-id: ${{ inputs.aws-access-key-id }} aws-access-key-id: ${{ inputs.aws-access-key-id }}
aws-secret-access-key: ${{ inputs.aws-secret-access-key }} aws-secret-access-key: ${{ inputs.aws-secret-access-key }}
- name: Configure ccache
if: inputs.ccache_enabled == 'true'
shell: bash
run: |
# Use ccache's default cache_dir (~/.ccache) - don't override it
# This avoids tilde expansion issues when setting it explicitly
# Create cache directory using ccache's default
mkdir -p ~/.ccache
# Configure ccache settings (but NOT cache_dir - use default)
# This overwrites any cached config to ensure fresh configuration
ccache --set-config=max_size=${{ inputs.ccache_max_size }}
ccache --set-config=hash_dir=${{ inputs.ccache_hash_dir }}
ccache --set-config=compiler_check=${{ inputs.ccache_compiler_check }}
# Note: Not setting CCACHE_DIR - let ccache use its default (~/.ccache)
# Print config for verification
echo "=== ccache configuration ==="
ccache -p
# Zero statistics before the build
ccache -z
- name: Configure project - name: Configure project
shell: bash shell: bash
run: | run: |
@@ -157,31 +194,22 @@ runs:
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=${TOOLCHAIN_FILE} \ -DCMAKE_TOOLCHAIN_FILE:FILEPATH=${TOOLCHAIN_FILE} \
-DCMAKE_BUILD_TYPE=${{ inputs.configuration }} -DCMAKE_BUILD_TYPE=${{ inputs.configuration }}
- name: Show ccache config before build
if: inputs.ccache_enabled == 'true'
shell: bash
run: |
echo "=========================================="
echo "ccache configuration before build"
echo "=========================================="
ccache -p
echo ""
- name: Build project - name: Build project
shell: bash shell: bash
run: | run: |
cd ${{ inputs.build_dir }} cd ${{ inputs.build_dir }}
# TEMPORARY: Add -v to see compile commands for ccache debugging
cmake --build . --config ${{ inputs.configuration }} --parallel $(nproc) -- -v cmake --build . --config ${{ inputs.configuration }} --parallel $(nproc) -- -v
- name: Debug ccache directory
if: inputs.ccache_enabled == 'true'
shell: bash
run: |
echo "=== ccache directory contents ==="
ls -laR ~/.ccache || echo "Directory doesn't exist"
echo ""
echo "=== Disk space ==="
df -h ~
echo ""
echo "=== ccache config ==="
ccache --show-config | head -30
echo ""
echo "=== Directory sizes ==="
du -sh ~/.ccache 2>/dev/null || echo "No directory"
find ~/.ccache -type f -name "*.conf" -o -type f ! -name "*.conf" 2>/dev/null | head -20 || true
echo ""
- name: Show ccache statistics - name: Show ccache statistics
if: inputs.ccache_enabled == 'true' if: inputs.ccache_enabled == 'true'
shell: bash shell: bash

View File

@@ -81,7 +81,6 @@ runs:
restore-keys: | restore-keys: |
${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}- ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}-
${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}- ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-
use-deltas: 'false'
aws-access-key-id: ${{ inputs.aws-access-key-id }} aws-access-key-id: ${{ inputs.aws-access-key-id }}
aws-secret-access-key: ${{ inputs.aws-secret-access-key }} aws-secret-access-key: ${{ inputs.aws-secret-access-key }}
@@ -166,6 +165,5 @@ runs:
with: with:
path: ~/.conan2 path: ~/.conan2
key: ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}-${{ inputs.configuration }} key: ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}-${{ inputs.configuration }}
use-deltas: 'false'
aws-access-key-id: ${{ inputs.aws-access-key-id }} aws-access-key-id: ${{ inputs.aws-access-key-id }}
aws-secret-access-key: ${{ inputs.aws-secret-access-key }} aws-secret-access-key: ${{ inputs.aws-secret-access-key }}

View File

@@ -78,14 +78,6 @@ jobs:
- name: Install ccache - name: Install ccache
run: brew install ccache run: brew install ccache
- name: Configure ccache
uses: ./.github/actions/xahau-configure-ccache
with:
max_size: 2G
hash_dir: true
compiler_check: content
is_main_branch: ${{ github.ref_name == env.MAIN_BRANCH_NAME }}
- name: Check environment - name: Check environment
run: | run: |
echo "PATH:" echo "PATH:"
@@ -130,6 +122,9 @@ jobs:
compiler-id: clang compiler-id: clang
cache_version: ${{ env.CACHE_VERSION }} cache_version: ${{ env.CACHE_VERSION }}
main_branch: ${{ env.MAIN_BRANCH_NAME }} main_branch: ${{ env.MAIN_BRANCH_NAME }}
stdlib: libcxx
aws-access-key-id: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
aws-secret-access-key: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
- name: Test - name: Test
run: | run: |

View File

@@ -231,13 +231,6 @@ jobs:
# Install Conan 2 # Install Conan 2
pip install --upgrade "conan>=2.0,<3" pip install --upgrade "conan>=2.0,<3"
- name: Configure ccache
uses: ./.github/actions/xahau-configure-ccache
with:
max_size: 2G
hash_dir: true
compiler_check: content
- name: Check environment - name: Check environment
run: | run: |
echo "PATH:" echo "PATH:"