fix: don't override ccache's default cache_dir

PROBLEM: Setting cache_dir='~/.ccache' explicitly stores the LITERAL tilde in ccache.conf, which might not expand properly. SOLUTION: Don't set cache_dir at all - let ccache use its default - Linux default: ~/.ccache - ccache will expand the tilde correctly when using default - Only configure max_size, hash_dir, compiler_check - Remove CCACHE_DIR env var (not needed with default) This should fix the issue where ccache runs but creates no cache files.
debug: add ccache directory inspection after build
2025-11-30 07:25:50 +00:00 · 2025-10-30 17:58:08 +07:00 · 2025-10-30 17:53:33 +07:00 · 2025-10-30 17:04:00 +07:00 · 2025-10-30 16:41:43 +07:00 · 2025-10-30 16:17:40 +07:00
17 changed files with 1288 additions and 557 deletions
--- a/.github/actions/xahau-actions-cache-restore/action.yml
+++ b/.github/actions/xahau-actions-cache-restore/action.yml
@@ -0,0 +1,328 @@
+name: 'Xahau Cache Restore (S3 + OverlayFS)'
+description: 'Drop-in replacement for actions/cache/restore using S3 and OverlayFS for delta caching'
+
+inputs:
+  path:
+    description: 'A list of files, directories, and wildcard patterns to cache (currently only single path supported)'
+    required: true
+  key:
+    description: 'An explicit key for restoring the cache'
+    required: true
+  restore-keys:
+    description: 'An ordered list of prefix-matched keys to use for restoring stale cache if no cache hit occurred for key'
+    required: false
+    default: ''
+  s3-bucket:
+    description: 'S3 bucket name for cache storage'
+    required: false
+    default: 'xahaud-github-actions-cache-niq'
+  s3-region:
+    description: 'S3 region'
+    required: false
+    default: 'us-east-1'
+  fail-on-cache-miss:
+    description: 'Fail the workflow if cache entry is not found'
+    required: false
+    default: 'false'
+  lookup-only:
+    description: 'Check if a cache entry exists for the given input(s) without downloading it'
+    required: false
+    default: 'false'
+  use-deltas:
+    description: 'Enable delta caching (download/upload incremental changes). Set to false for base-only caching.'
+    required: false
+    default: 'true'
+  # Note: Composite actions can't access secrets.* directly - must be passed from workflow
+  aws-access-key-id:
+    description: 'AWS Access Key ID for S3 access'
+    required: true
+  aws-secret-access-key:
+    description: 'AWS Secret Access Key for S3 access'
+    required: true
+
+outputs:
+  cache-hit:
+    description: 'A boolean value to indicate an exact match was found for the primary key'
+    value: ${{ steps.restore-cache.outputs.cache-hit }}
+  cache-primary-key:
+    description: 'The key that was used to restore the cache (may be from restore-keys)'
+    value: ${{ steps.restore-cache.outputs.cache-primary-key }}
+  cache-matched-key:
+    description: 'The key that matched (same as cache-primary-key for compatibility)'
+    value: ${{ steps.restore-cache.outputs.cache-primary-key }}
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Restore cache from S3 with OverlayFS
+      id: restore-cache
+      shell: bash
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.aws-secret-access-key }}
+        S3_BUCKET: ${{ inputs.s3-bucket }}
+        S3_REGION: ${{ inputs.s3-region }}
+        CACHE_KEY: ${{ inputs.key }}
+        RESTORE_KEYS: ${{ inputs.restore-keys }}
+        TARGET_PATH: ${{ inputs.path }}
+        FAIL_ON_MISS: ${{ inputs.fail-on-cache-miss }}
+        LOOKUP_ONLY: ${{ inputs.lookup-only }}
+        USE_DELTAS: ${{ inputs.use-deltas }}
+        COMMIT_MSG: ${{ github.event.head_commit.message }}
+      run: |
+        set -euo pipefail
+
+        echo "=========================================="
+        echo "Xahau Cache Restore (S3 + OverlayFS)"
+        echo "=========================================="
+        echo "Target path: ${TARGET_PATH}"
+        echo "Primary key: ${CACHE_KEY}"
+        echo "S3 bucket: s3://${S3_BUCKET}"
+        echo "Use deltas: ${USE_DELTAS}"
+        echo ""
+
+        # Normalize target path (expand tilde and resolve to absolute path)
+        # This ensures consistent path comparison in the mount registry
+        if [[ "${TARGET_PATH}" == ~* ]]; then
+          # Expand tilde manually (works even if directory doesn't exist yet)
+          TARGET_PATH="${HOME}${TARGET_PATH:1}"
+        fi
+        echo "Normalized target path: ${TARGET_PATH}"
+
+        # Generate unique cache workspace
+        CACHE_HASH=$(echo "${CACHE_KEY}" | md5sum | cut -d' ' -f1)
+        CACHE_WORKSPACE="/tmp/xahau-cache-${CACHE_HASH}"
+
+        echo "Cache workspace: ${CACHE_WORKSPACE}"
+
+        # Check for [ci-clear-cache] tag in commit message
+        if echo "${COMMIT_MSG}" | grep -q '\[ci-clear-cache\]'; then
+          echo ""
+          echo "🗑️  [ci-clear-cache] detected in commit message"
+          echo "Clearing cache for key: ${CACHE_KEY}"
+          echo ""
+
+          # Delete base layer
+          S3_BASE_KEY="s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst"
+          if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
+            echo "Deleting base layer: ${S3_BASE_KEY}"
+            aws s3 rm "${S3_BASE_KEY}" --region "${S3_REGION}" 2>/dev/null || true
+            echo "✓ Base layer deleted"
+          else
+            echo "ℹ️  No base layer found to delete"
+          fi
+
+          # Delete all delta layers for this key
+          echo "Deleting all delta layers matching: ${CACHE_KEY}-delta-*"
+          DELTA_COUNT=$(aws s3 ls "s3://${S3_BUCKET}/" --region "${S3_REGION}" | grep "${CACHE_KEY}-delta-" | wc -l || echo "0")
+          DELTA_COUNT=$(echo "${DELTA_COUNT}" | tr -d ' \n')  # Trim whitespace
+          if [ "${DELTA_COUNT}" -gt 0 ]; then
+            aws s3 rm "s3://${S3_BUCKET}/" --recursive \
+              --exclude "*" \
+              --include "${CACHE_KEY}-delta-*" \
+              --region "${S3_REGION}" 2>/dev/null || true
+            echo "✓ Deleted ${DELTA_COUNT} delta layer(s)"
+          else
+            echo "ℹ️  No delta layers found to delete"
+          fi
+
+          echo ""
+          echo "✅ Cache cleared successfully"
+          echo "Build will proceed from scratch (bootstrap mode)"
+          echo ""
+        fi
+
+        # Create OverlayFS directory structure
+        mkdir -p "${CACHE_WORKSPACE}"/{base,upper,work,merged}
+
+        # Function to try downloading from S3
+        try_restore_key() {
+          local try_key="$1"
+          local s3_base="s3://${S3_BUCKET}/${try_key}-base.tar.zst"
+
+          echo "Trying cache key: ${try_key}"
+
+          # Check if base exists (one base per key, immutable)
+          echo "Checking for base layer..."
+          if aws s3 ls "${s3_base}" --region "${S3_REGION}" >/dev/null 2>&1; then
+            echo "✓ Found base layer: ${s3_base}"
+
+            if [ "${LOOKUP_ONLY}" = "true" ]; then
+              echo "Lookup-only mode: cache exists, skipping download"
+              return 0
+            fi
+
+            # Download base layer
+            echo "Downloading base layer..."
+            aws s3 cp "${s3_base}" /tmp/cache-base.tar.zst --region "${S3_REGION}" --quiet
+
+            # Extract base layer
+            echo "Extracting base layer..."
+            tar -xf /tmp/cache-base.tar.zst -C "${CACHE_WORKSPACE}/base"
+            rm /tmp/cache-base.tar.zst
+
+            # Query for latest timestamped delta (only if use-deltas enabled)
+            if [ "${USE_DELTAS}" = "true" ]; then
+              echo "Querying for latest delta..."
+              LATEST_DELTA=$(aws s3api list-objects-v2 \
+                --bucket "${S3_BUCKET}" \
+                --prefix "${try_key}-delta-" \
+                --region "${S3_REGION}" \
+                --query 'sort_by(Contents, &LastModified)[-1].Key' \
+                --output text 2>/dev/null || echo "")
+
+              if [ -n "${LATEST_DELTA}" ] && [ "${LATEST_DELTA}" != "None" ]; then
+                echo "✓ Found latest delta: ${LATEST_DELTA}"
+                echo "Downloading delta layer..."
+                aws s3 cp "s3://${S3_BUCKET}/${LATEST_DELTA}" /tmp/cache-delta.tar.zst --region "${S3_REGION}" --quiet
+
+                echo "Extracting delta layer..."
+                tar -xf /tmp/cache-delta.tar.zst -C "${CACHE_WORKSPACE}/upper" 2>/dev/null || true
+                rm /tmp/cache-delta.tar.zst
+              else
+                echo "ℹ No delta layer found (this is fine for first build)"
+              fi
+            else
+              echo "ℹ Delta caching disabled (use-deltas: false)"
+            fi
+
+            return 0
+          else
+            echo "✗ No base layer found for key: ${try_key}"
+            return 1
+          fi
+        }
+
+        # Try primary key first
+        MATCHED_KEY=""
+        EXACT_MATCH="false"
+
+        if try_restore_key "${CACHE_KEY}"; then
+          MATCHED_KEY="${CACHE_KEY}"
+          EXACT_MATCH="true"
+          echo ""
+          echo "🎯 Exact cache hit for key: ${CACHE_KEY}"
+        else
+          # Try restore-keys (prefix matching)
+          if [ -n "${RESTORE_KEYS}" ]; then
+            echo ""
+            echo "Primary key not found, trying restore-keys..."
+
+            # Split restore-keys by newline
+            while IFS= read -r restore_key; do
+              # Skip empty lines
+              [ -z "${restore_key}" ] && continue
+
+              # Trim whitespace
+              restore_key=$(echo "${restore_key}" | xargs)
+
+              if try_restore_key "${restore_key}"; then
+                MATCHED_KEY="${restore_key}"
+                EXACT_MATCH="false"
+                echo ""
+                echo "✓ Cache restored from fallback key: ${restore_key}"
+                break
+              fi
+            done <<< "${RESTORE_KEYS}"
+          fi
+        fi
+
+        # Check if we found anything
+        if [ -z "${MATCHED_KEY}" ]; then
+          echo ""
+          echo "❌ No cache found for key: ${CACHE_KEY}"
+          echo "This is BOOTSTRAP mode - first build for this cache key"
+
+          if [ "${FAIL_ON_MISS}" = "true" ]; then
+            echo "fail-on-cache-miss is enabled, failing workflow"
+            exit 1
+          fi
+
+          # Set outputs for cache miss
+          echo "cache-hit=false" >> $GITHUB_OUTPUT
+          echo "cache-primary-key=" >> $GITHUB_OUTPUT
+
+          # Create empty cache directory for bootstrap
+          mkdir -p "${TARGET_PATH}"
+
+          # Record bootstrap mode for save action
+          # Format: path:workspace:matched_key:primary_key:exact_match:use_deltas
+          # For bootstrap: workspace="bootstrap", matched_key=primary_key, exact_match=false
+          MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt"
+          echo "${TARGET_PATH}:bootstrap:${CACHE_KEY}:${CACHE_KEY}:false:${USE_DELTAS}" >> "${MOUNT_REGISTRY}"
+
+          echo ""
+          echo "=========================================="
+          echo "Cache restore completed (bootstrap mode)"
+          echo "Created empty cache directory: ${TARGET_PATH}"
+          echo "=========================================="
+          exit 0
+        fi
+
+        # If lookup-only, we're done
+        if [ "${LOOKUP_ONLY}" = "true" ]; then
+          echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
+          echo "cache-primary-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
+
+          # Clean up workspace
+          rm -rf "${CACHE_WORKSPACE}"
+
+          echo ""
+          echo "=========================================="
+          echo "Cache lookup completed (lookup-only mode)"
+          echo "=========================================="
+          exit 0
+        fi
+
+        # Mount OverlayFS
+        echo ""
+        echo "Mounting OverlayFS..."
+        sudo mount -t overlay overlay \
+          -o lowerdir="${CACHE_WORKSPACE}/base",upperdir="${CACHE_WORKSPACE}/upper",workdir="${CACHE_WORKSPACE}/work" \
+          "${CACHE_WORKSPACE}/merged"
+
+        # Verify mount
+        if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
+          echo "✓ OverlayFS mounted successfully"
+        else
+          echo "❌ Failed to mount OverlayFS"
+          exit 1
+        fi
+
+        # Create target directory parent if needed
+        TARGET_PARENT=$(dirname "${TARGET_PATH}")
+        mkdir -p "${TARGET_PARENT}"
+
+        # Remove existing target if it exists
+        if [ -e "${TARGET_PATH}" ]; then
+          echo "Removing existing target: ${TARGET_PATH}"
+          rm -rf "${TARGET_PATH}"
+        fi
+
+        # Symlink target path to merged view
+        echo "Creating symlink: ${TARGET_PATH} -> ${CACHE_WORKSPACE}/merged"
+        ln -s "${CACHE_WORKSPACE}/merged" "${TARGET_PATH}"
+
+        # Save mount info for cleanup/save later
+        # Format: path:workspace:matched_key:primary_key:exact_match:use_deltas
+        # This tells save action whether to create new base (partial match) or just delta (exact match)
+        MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt"
+        echo "${TARGET_PATH}:${CACHE_WORKSPACE}:${MATCHED_KEY}:${CACHE_KEY}:${EXACT_MATCH}:${USE_DELTAS}" >> "${MOUNT_REGISTRY}"
+
+        # Set outputs
+        echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
+        echo "cache-primary-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
+
+        # Show statistics
+        echo ""
+        echo "Cache statistics:"
+        echo "  Base layer size:  $(du -sh ${CACHE_WORKSPACE}/base 2>/dev/null | cut -f1 || echo '0')"
+        echo "  Delta layer size: $(du -sh ${CACHE_WORKSPACE}/upper 2>/dev/null | cut -f1 || echo '0')"
+        echo "  Merged view size: $(du -sh ${CACHE_WORKSPACE}/merged 2>/dev/null | cut -f1 || echo '0')"
+
+        echo ""
+        echo "=========================================="
+        echo "Cache restore completed successfully"
+        echo "Exact match: ${EXACT_MATCH}"
+        echo "Matched key: ${MATCHED_KEY}"
+        echo "=========================================="
--- a/.github/actions/xahau-actions-cache-save/action.yml
+++ b/.github/actions/xahau-actions-cache-save/action.yml
@@ -0,0 +1,398 @@
+name: 'Xahau Cache Save (S3 + OverlayFS)'
+description: 'Drop-in replacement for actions/cache/save using S3 and OverlayFS for delta caching'
+
+inputs:
+  path:
+    description: 'A list of files, directories, and wildcard patterns to cache (currently only single path supported)'
+    required: true
+  key:
+    description: 'An explicit key for saving the cache'
+    required: true
+  s3-bucket:
+    description: 'S3 bucket name for cache storage'
+    required: false
+    default: 'xahaud-github-actions-cache-niq'
+  s3-region:
+    description: 'S3 region'
+    required: false
+    default: 'us-east-1'
+  use-deltas:
+    description: 'Enable delta caching (download/upload incremental changes). Set to false for base-only caching.'
+    required: false
+    default: 'true'
+  # Note: Composite actions can't access secrets.* directly - must be passed from workflow
+  aws-access-key-id:
+    description: 'AWS Access Key ID for S3 access'
+    required: true
+  aws-secret-access-key:
+    description: 'AWS Secret Access Key for S3 access'
+    required: true
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Save cache to S3 with OverlayFS delta
+      shell: bash
+      env:
+        AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
+        AWS_SECRET_ACCESS_KEY: ${{ inputs.aws-secret-access-key }}
+        S3_BUCKET: ${{ inputs.s3-bucket }}
+        S3_REGION: ${{ inputs.s3-region }}
+        CACHE_KEY: ${{ inputs.key }}
+        TARGET_PATH: ${{ inputs.path }}
+        USE_DELTAS: ${{ inputs.use-deltas }}
+      run: |
+        set -euo pipefail
+
+        echo "=========================================="
+        echo "Xahau Cache Save (S3 + OverlayFS)"
+        echo "=========================================="
+        echo "Target path: ${TARGET_PATH}"
+        echo "Cache key: ${CACHE_KEY}"
+        echo "S3 bucket: s3://${S3_BUCKET}"
+        echo ""
+
+        # Normalize target path (expand tilde and resolve to absolute path)
+        # This ensures consistent path comparison with the mount registry
+        if [[ "${TARGET_PATH}" == ~* ]]; then
+          # Expand tilde manually (works even if directory doesn't exist yet)
+          TARGET_PATH="${HOME}${TARGET_PATH:1}"
+        fi
+        echo "Normalized target path: ${TARGET_PATH}"
+        echo ""
+
+        # Find the cache workspace from mount registry
+        MOUNT_REGISTRY="/tmp/xahau-cache-mounts.txt"
+
+        if [ ! -f "${MOUNT_REGISTRY}" ]; then
+          echo "⚠️  No cache mounts found (mount registry doesn't exist)"
+          echo "This usually means cache restore was not called, or there was no cache to restore."
+          echo "Skipping cache save."
+          exit 0
+        fi
+
+        # Find entry for this path
+        # Format: path:workspace:matched_key:primary_key:exact_match:use_deltas
+        # Bootstrap mode: path:bootstrap:key:key:false:true/false (workspace="bootstrap")
+        CACHE_WORKSPACE=""
+        MATCHED_KEY=""
+        PRIMARY_KEY=""
+        EXACT_MATCH=""
+        REGISTRY_USE_DELTAS=""
+
+        while IFS=: read -r mount_path mount_workspace mount_matched_key mount_primary_key mount_exact_match mount_use_deltas; do
+          if [ "${mount_path}" = "${TARGET_PATH}" ]; then
+            CACHE_WORKSPACE="${mount_workspace}"
+            MATCHED_KEY="${mount_matched_key}"
+            PRIMARY_KEY="${mount_primary_key}"
+            EXACT_MATCH="${mount_exact_match}"
+            REGISTRY_USE_DELTAS="${mount_use_deltas}"
+            break
+          fi
+        done < "${MOUNT_REGISTRY}"
+
+        if [ -z "${CACHE_WORKSPACE}" ] && [ -z "${MATCHED_KEY}" ]; then
+          echo "⚠️  No cache entry found for path: ${TARGET_PATH}"
+          echo "This usually means cache restore was not called for this path."
+          echo "Skipping cache save."
+          exit 0
+        fi
+
+        # Determine cache mode
+        if [ "${CACHE_WORKSPACE}" = "bootstrap" ]; then
+          CACHE_MODE="bootstrap"
+          PRIMARY_KEY="${MATCHED_KEY}"  # In bootstrap, matched_key field contains primary key
+          echo "Cache mode: BOOTSTRAP (first build for this key)"
+          echo "Primary key: ${PRIMARY_KEY}"
+        elif [ "${EXACT_MATCH}" = "false" ]; then
+          CACHE_MODE="partial-match"
+          echo "Cache mode: PARTIAL MATCH (restore-key used)"
+          echo "Cache workspace: ${CACHE_WORKSPACE}"
+          echo "Matched key from restore: ${MATCHED_KEY}"
+          echo "Primary key (will save new base): ${PRIMARY_KEY}"
+        else
+          CACHE_MODE="exact-match"
+          echo "Cache mode: EXACT MATCH (cache hit)"
+          echo "Cache workspace: ${CACHE_WORKSPACE}"
+          echo "Matched key: ${MATCHED_KEY}"
+        fi
+        echo "Use deltas: ${REGISTRY_USE_DELTAS}"
+        echo ""
+
+        # Handle different cache modes
+        if [ "${CACHE_MODE}" = "bootstrap" ]; then
+          # Bootstrap: Save entire cache as base layer (no OverlayFS was used)
+          echo "Bootstrap mode: Creating initial base layer from ${TARGET_PATH}"
+
+          BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst"
+          echo "Creating base tarball..."
+          tar -cf - -C "${TARGET_PATH}" . | zstd -3 -T0 -q -o "${BASE_TARBALL}"
+
+          BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1)
+          echo "✓ Base tarball created: ${BASE_SIZE}"
+          echo ""
+
+          # Use static base name (one base per key, immutable)
+          S3_BASE_KEY="s3://${S3_BUCKET}/${PRIMARY_KEY}-base.tar.zst"
+
+          # Check if base already exists (immutability - first write wins)
+          if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
+            echo "⚠️  Base layer already exists: ${S3_BASE_KEY}"
+            echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
+          else
+            echo "Uploading base layer to S3..."
+            echo "  Key: ${PRIMARY_KEY}-base.tar.zst"
+
+            aws s3api put-object \
+              --bucket "${S3_BUCKET}" \
+              --key "${PRIMARY_KEY}-base.tar.zst" \
+              --body "${BASE_TARBALL}" \
+              --tagging 'type=base' \
+              --region "${S3_REGION}" \
+              >/dev/null
+
+            echo "✓ Uploaded: ${S3_BASE_KEY}"
+          fi
+
+          # Cleanup
+          rm -f "${BASE_TARBALL}"
+
+          echo ""
+          echo "=========================================="
+          echo "Bootstrap cache save completed"
+          echo "Base size: ${BASE_SIZE}"
+          echo "Cache key: ${PRIMARY_KEY}"
+          echo "=========================================="
+          exit 0
+
+        elif [ "${CACHE_MODE}" = "partial-match" ]; then
+          # Partial match: Save merged view as new base ONLY (no delta)
+          # The delta is relative to the OLD base, not the NEW base we're creating
+          echo "Partial match mode: Saving new base layer for primary key"
+          echo "Note: Delta will NOT be saved (it's relative to old base)"
+
+          BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst"
+          echo "Creating base tarball from merged view..."
+          tar -cf - -C "${CACHE_WORKSPACE}/merged" . | zstd -3 -T0 -q -o "${BASE_TARBALL}"
+
+          BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1)
+          echo "✓ Base tarball created: ${BASE_SIZE}"
+          echo ""
+
+          # Use static base name (one base per key, immutable)
+          S3_BASE_KEY="s3://${S3_BUCKET}/${PRIMARY_KEY}-base.tar.zst"
+
+          # Check if base already exists (immutability - first write wins)
+          if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
+            echo "⚠️  Base layer already exists: ${S3_BASE_KEY}"
+            echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
+          else
+            echo "Uploading new base layer to S3..."
+            echo "  Key: ${PRIMARY_KEY}-base.tar.zst"
+
+            aws s3api put-object \
+              --bucket "${S3_BUCKET}" \
+              --key "${PRIMARY_KEY}-base.tar.zst" \
+              --body "${BASE_TARBALL}" \
+              --tagging 'type=base' \
+              --region "${S3_REGION}" \
+              >/dev/null
+
+            echo "✓ Uploaded: ${S3_BASE_KEY}"
+          fi
+
+          # Cleanup
+          rm -f "${BASE_TARBALL}"
+
+          # Unmount and cleanup
+          echo ""
+          echo "Cleaning up..."
+          if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
+            sudo umount "${CACHE_WORKSPACE}/merged" || {
+              echo "⚠️  Warning: Failed to unmount ${CACHE_WORKSPACE}/merged"
+              echo "Attempting lazy unmount..."
+              sudo umount -l "${CACHE_WORKSPACE}/merged" || true
+            }
+          fi
+          rm -rf "${CACHE_WORKSPACE}"
+
+          # Remove from registry
+          if [ -f "${MOUNT_REGISTRY}" ]; then
+            grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
+            mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
+          fi
+
+          echo "✓ Cleanup completed"
+
+          echo ""
+          echo "=========================================="
+          echo "Partial match cache save completed"
+          echo "New base created for: ${PRIMARY_KEY}"
+          echo "Base size: ${BASE_SIZE}"
+          if [ "${REGISTRY_USE_DELTAS}" = "true" ]; then
+            echo "Next exact-match build will create deltas from this base"
+          else
+            echo "Next exact-match build will reuse this base (base-only mode)"
+          fi
+          echo "=========================================="
+          exit 0
+        fi
+
+        # For exact-match ONLY: Save delta (if use-deltas enabled)
+        if [ "${CACHE_MODE}" = "exact-match" ]; then
+          # If deltas are disabled, just cleanup and exit
+          if [ "${REGISTRY_USE_DELTAS}" != "true" ]; then
+            echo "ℹ️  Delta caching disabled (use-deltas: false)"
+            echo "Base already exists for this key, nothing to save."
+
+            # Unmount and cleanup
+            echo ""
+            echo "Cleaning up..."
+            if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
+              sudo umount "${CACHE_WORKSPACE}/merged" 2>/dev/null || true
+            fi
+            rm -rf "${CACHE_WORKSPACE}"
+
+            # Remove from registry
+            if [ -f "${MOUNT_REGISTRY}" ]; then
+              grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
+              mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
+            fi
+
+            echo ""
+            echo "=========================================="
+            echo "Cache save completed (base-only mode)"
+            echo "=========================================="
+            exit 0
+          fi
+
+          # Check if upper layer has any changes
+          if [ -z "$(ls -A ${CACHE_WORKSPACE}/upper 2>/dev/null)" ]; then
+            echo "ℹ️  No changes detected in upper layer (cache is unchanged)"
+            echo "Skipping delta upload to save bandwidth."
+
+            # Still unmount and cleanup
+            echo ""
+            echo "Cleaning up..."
+            sudo umount "${CACHE_WORKSPACE}/merged" 2>/dev/null || true
+            rm -rf "${CACHE_WORKSPACE}"
+
+            echo ""
+            echo "=========================================="
+            echo "Cache save completed (no changes)"
+            echo "=========================================="
+            exit 0
+          fi
+
+          # Show delta statistics
+          echo "Delta layer statistics:"
+          echo "  Files changed: $(find ${CACHE_WORKSPACE}/upper -type f 2>/dev/null | wc -l)"
+          echo "  Delta size:    $(du -sh ${CACHE_WORKSPACE}/upper 2>/dev/null | cut -f1)"
+          echo ""
+
+          # Create delta tarball from upper layer
+          echo "Creating delta tarball..."
+          DELTA_TARBALL="/tmp/xahau-cache-delta-$$.tar.zst"
+
+          tar -cf - -C "${CACHE_WORKSPACE}/upper" . | zstd -3 -T0 -q -o "${DELTA_TARBALL}"
+
+          DELTA_SIZE=$(du -h "${DELTA_TARBALL}" | cut -f1)
+          echo "✓ Delta tarball created: ${DELTA_SIZE}"
+          echo ""
+
+          # Upload timestamped delta (no overwrites = zero concurrency issues)
+          TIMESTAMP=$(date +%Y%m%d%H%M%S)
+          COMMIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
+
+          # Use PRIMARY_KEY for delta (ensures deltas match their base)
+          S3_DELTA_TIMESTAMPED="s3://${S3_BUCKET}/${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst"
+
+          echo "Uploading timestamped delta to S3..."
+          echo "  Key: ${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst"
+
+          # Upload with tag (deltas cleaned up inline - keep last 1)
+          aws s3api put-object \
+            --bucket "${S3_BUCKET}" \
+            --key "${PRIMARY_KEY}-delta-${TIMESTAMP}-${COMMIT_SHA}.tar.zst" \
+            --body "${DELTA_TARBALL}" \
+            --tagging 'type=delta-archive' \
+            --region "${S3_REGION}" \
+            >/dev/null
+
+          echo "✓ Uploaded: ${S3_DELTA_TIMESTAMPED}"
+
+          # Inline cleanup: Keep only latest delta (the one we just uploaded)
+          echo ""
+          echo "Cleaning up old deltas (keeping only latest)..."
+
+          # List all deltas for this key, sorted by LastModified (oldest first)
+          ALL_DELTAS=$(aws s3api list-objects-v2 \
+            --bucket "${S3_BUCKET}" \
+            --prefix "${PRIMARY_KEY}-delta-" \
+            --region "${S3_REGION}" \
+            --query 'sort_by(Contents, &LastModified)[*].Key' \
+            --output json 2>/dev/null || echo "[]")
+
+          DELTA_COUNT=$(echo "${ALL_DELTAS}" | jq 'length' 2>/dev/null || echo "0")
+
+          if [ "${DELTA_COUNT}" -gt 1 ]; then
+            # Keep last 1 (newest), delete all older ones (all except last 1 = [0:-1])
+            OLD_DELTAS=$(echo "${ALL_DELTAS}" | jq -r '.[0:-1][]' 2>/dev/null)
+
+            if [ -n "${OLD_DELTAS}" ]; then
+              DELETE_COUNT=$((DELTA_COUNT - 1))
+              echo "  Found ${DELETE_COUNT} old delta(s) to delete"
+
+              # Create delete batch request JSON
+              DELETE_OBJECTS=$(echo "${OLD_DELTAS}" | jq -R -s -c 'split("\n") | map(select(length > 0)) | map({Key: .}) | {Objects: ., Quiet: true}' 2>/dev/null)
+
+              if [ -n "${DELETE_OBJECTS}" ]; then
+                aws s3api delete-objects \
+                  --bucket "${S3_BUCKET}" \
+                  --delete "${DELETE_OBJECTS}" \
+                  --region "${S3_REGION}" \
+                  >/dev/null 2>&1
+
+                echo "✓ Deleted ${DELETE_COUNT} old delta(s)"
+              fi
+            fi
+          else
+            echo "ℹ️  Only ${DELTA_COUNT} delta(s) exist, no cleanup needed"
+          fi
+
+          # Cleanup delta tarball
+          rm -f "${DELTA_TARBALL}"
+
+          # Cleanup: Unmount OverlayFS and remove workspace
+          echo ""
+          echo "Cleaning up..."
+
+          if mount | grep -q "${CACHE_WORKSPACE}/merged"; then
+            sudo umount "${CACHE_WORKSPACE}/merged" || {
+              echo "⚠️  Warning: Failed to unmount ${CACHE_WORKSPACE}/merged"
+              echo "Attempting lazy unmount..."
+              sudo umount -l "${CACHE_WORKSPACE}/merged" || true
+            }
+          fi
+
+          # Remove workspace
+          rm -rf "${CACHE_WORKSPACE}"
+        fi
+
+        # Remove from registry
+        if [ -f "${MOUNT_REGISTRY}" ]; then
+          grep -v "^${TARGET_PATH}:" "${MOUNT_REGISTRY}" > "${MOUNT_REGISTRY}.tmp" 2>/dev/null || true
+          mv "${MOUNT_REGISTRY}.tmp" "${MOUNT_REGISTRY}" 2>/dev/null || true
+        fi
+
+        echo "✓ Cleanup completed"
+
+        echo ""
+        echo "=========================================="
+        echo "Cache save completed successfully"
+        echo "Mode: ${CACHE_MODE}"
+        echo "Cache key: ${PRIMARY_KEY}"
+        if [ -n "${DELTA_SIZE:-}" ]; then
+          echo "Delta size: ${DELTA_SIZE}"
+        fi
+        echo "=========================================="
--- a/.github/actions/xahau-configure-ccache/action.yml
+++ b/.github/actions/xahau-configure-ccache/action.yml
@@ -0,0 +1,45 @@
+name: 'Configure ccache'
+description: 'Sets up ccache with consistent configuration'
+
+inputs:
+  cache_dir:
+    description: 'Path to ccache directory'
+    required: false
+    default: '~/.ccache'
+  max_size:
+    description: 'Maximum cache size'
+    required: false
+    default: '2G'
+  hash_dir:
+    description: 'Whether to include directory paths in hash'
+    required: false
+    default: 'true'
+  compiler_check:
+    description: 'How to check compiler for changes'
+    required: false
+    default: 'content'
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Configure ccache
+      shell: bash
+      run: |
+        # Use ccache's default cache_dir (~/.ccache) - don't override it
+        # This avoids tilde expansion issues when setting it explicitly
+
+        # Create cache directory using ccache's default
+        mkdir -p ~/.ccache
+
+        # Configure ccache settings (but NOT cache_dir - use default)
+        ccache --set-config=max_size=${{ inputs.max_size }}
+        ccache --set-config=hash_dir=${{ inputs.hash_dir }}
+        ccache --set-config=compiler_check=${{ inputs.compiler_check }}
+
+        # Note: Not setting CCACHE_DIR - let ccache use its default (~/.ccache)
+
+        # Print config for verification
+        ccache -p
+
+        # Zero statistics before the build
+        ccache -z
--- a/.github/actions/xahau-ga-build/action.yml
+++ b/.github/actions/xahau-ga-build/action.yml
@@ -47,18 +47,6 @@ inputs:
    description: 'GCC version to use for Clang toolchain (e.g. 11, 13)'
    required: false
    default: ''
-  ccache_max_size:
-    description: 'Maximum ccache size'
-    required: false
-    default: '2G'
-  ccache_hash_dir:
-    description: 'Whether to include directory paths in hash'
-    required: false
-    default: 'true'
-  ccache_compiler_check:
-    description: 'How to check compiler for changes'
-    required: false
-    default: 'content'
  aws-access-key-id:
    description: 'AWS Access Key ID for S3 cache storage'
    required: true
@@ -80,7 +68,7 @@ runs:
    - name: Restore ccache directory
      if: inputs.ccache_enabled == 'true'
      id: ccache-restore
-      uses: ./.github/actions/xahau-ga-cache-restore
+      uses: ./.github/actions/xahau-actions-cache-restore
      with:
        path: ~/.ccache
        key: ${{ runner.os }}-ccache-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ inputs.configuration }}-${{ steps.safe-branch.outputs.name }}
@@ -91,31 +79,6 @@ runs:
        aws-access-key-id: ${{ inputs.aws-access-key-id }}
        aws-secret-access-key: ${{ inputs.aws-secret-access-key }}

-    - name: Configure ccache
-      if: inputs.ccache_enabled == 'true'
-      shell: bash
-      run: |
-        # Use ccache's default cache_dir (~/.ccache) - don't override it
-        # This avoids tilde expansion issues when setting it explicitly
-
-        # Create cache directory using ccache's default
-        mkdir -p ~/.ccache
-
-        # Configure ccache settings (but NOT cache_dir - use default)
-        # This overwrites any cached config to ensure fresh configuration
-        ccache --set-config=max_size=${{ inputs.ccache_max_size }}
-        ccache --set-config=hash_dir=${{ inputs.ccache_hash_dir }}
-        ccache --set-config=compiler_check=${{ inputs.ccache_compiler_check }}
-
-        # Note: Not setting CCACHE_DIR - let ccache use its default (~/.ccache)
-
-        # Print config for verification
-        echo "=== ccache configuration ==="
-        ccache -p
-
-        # Zero statistics before the build
-        ccache -z
-
    - name: Configure project
      shell: bash
      run: |
@@ -194,29 +157,30 @@ runs:
          -DCMAKE_TOOLCHAIN_FILE:FILEPATH=${TOOLCHAIN_FILE} \
          -DCMAKE_BUILD_TYPE=${{ inputs.configuration }}

-    - name: Show ccache config before build
-      if: inputs.ccache_enabled == 'true'
-      shell: bash
-      run: |
-        echo "=========================================="
-        echo "ccache configuration before build"
-        echo "=========================================="
-        ccache -p
-        echo ""
-
    - name: Build project
      shell: bash
      run: |
        cd ${{ inputs.build_dir }}
+        # TEMPORARY: Add -v to see compile commands for ccache debugging
+        cmake --build . --config ${{ inputs.configuration }} --parallel $(nproc) -- -v

-        # Check for verbose build flag in commit message
-        VERBOSE_FLAG=""
-        if echo "${XAHAU_GA_COMMIT_MSG}" | grep -q '\[ci-ga-cmake-verbose\]'; then
-          echo "🔊 [ci-ga-cmake-verbose] detected - enabling verbose output"
-          VERBOSE_FLAG="-- -v"
-        fi
-
-        cmake --build . --config ${{ inputs.configuration }} --parallel $(nproc) ${VERBOSE_FLAG}
+    - name: Debug ccache directory
+      if: inputs.ccache_enabled == 'true'
+      shell: bash
+      run: |
+        echo "=== ccache directory contents ==="
+        ls -laR ~/.ccache || echo "Directory doesn't exist"
+        echo ""
+        echo "=== Disk space ==="
+        df -h ~
+        echo ""
+        echo "=== ccache config ==="
+        ccache --show-config | head -30
+        echo ""
+        echo "=== Directory sizes ==="
+        du -sh ~/.ccache 2>/dev/null || echo "No directory"
+        find ~/.ccache -type f -name "*.conf" -o -type f ! -name "*.conf" 2>/dev/null | head -20 || true
+        echo ""

    - name: Show ccache statistics
      if: inputs.ccache_enabled == 'true'
@@ -224,8 +188,8 @@ runs:
      run: ccache -s

    - name: Save ccache directory
-      if: success() && inputs.ccache_enabled == 'true'
-      uses: ./.github/actions/xahau-ga-cache-save
+      if: always() && inputs.ccache_enabled == 'true'
+      uses: ./.github/actions/xahau-actions-cache-save
      with:
        path: ~/.ccache
        key: ${{ runner.os }}-ccache-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ inputs.configuration }}-${{ steps.safe-branch.outputs.name }}
--- a/.github/actions/xahau-ga-cache-restore/action.yml
+++ b/.github/actions/xahau-ga-cache-restore/action.yml
@@ -1,291 +0,0 @@
-name: 'Xahau Cache Restore (S3)'
-bump: 1
-description: 'Drop-in replacement for actions/cache/restore using S3 storage'
-
-inputs:
-  path:
-    description: 'A list of files, directories, and wildcard patterns to cache (currently only single path supported)'
-    required: true
-  key:
-    description: 'An explicit key for restoring the cache'
-    required: true
-  restore-keys:
-    description: 'An ordered list of prefix-matched keys to use for restoring stale cache if no cache hit occurred for key'
-    required: false
-    default: ''
-  s3-bucket:
-    description: 'S3 bucket name for cache storage'
-    required: false
-    default: 'xahaud-github-actions-cache-niq'
-  s3-region:
-    description: 'S3 region'
-    required: false
-    default: 'us-east-1'
-  fail-on-cache-miss:
-    description: 'Fail the workflow if cache entry is not found'
-    required: false
-    default: 'false'
-  lookup-only:
-    description: 'Check if a cache entry exists for the given input(s) without downloading it'
-    required: false
-    default: 'false'
-  # Note: Composite actions can't access secrets.* directly - must be passed from workflow
-  aws-access-key-id:
-    description: 'AWS Access Key ID for S3 access'
-    required: true
-  aws-secret-access-key:
-    description: 'AWS Secret Access Key for S3 access'
-    required: true
-
-outputs:
-  cache-hit:
-    description: 'A boolean value to indicate an exact match was found for the primary key'
-    value: ${{ steps.restore-cache.outputs.cache-hit }}
-  cache-primary-key:
-    description: 'The key that was used to restore the cache (may be from restore-keys)'
-    value: ${{ steps.restore-cache.outputs.cache-primary-key }}
-  cache-matched-key:
-    description: 'The key that was used to restore the cache (exact or prefix match)'
-    value: ${{ steps.restore-cache.outputs.cache-matched-key }}
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Restore cache from S3
-      id: restore-cache
-      shell: bash
-      env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.aws-secret-access-key }}
-        S3_BUCKET: ${{ inputs.s3-bucket }}
-        S3_REGION: ${{ inputs.s3-region }}
-        CACHE_KEY: ${{ inputs.key }}
-        RESTORE_KEYS: ${{ inputs.restore-keys }}
-        TARGET_PATH: ${{ inputs.path }}
-        FAIL_ON_MISS: ${{ inputs.fail-on-cache-miss }}
-        LOOKUP_ONLY: ${{ inputs.lookup-only }}
-      run: |
-        set -euo pipefail
-
-        echo "=========================================="
-        echo "Xahau Cache Restore (S3)"
-        echo "=========================================="
-        echo "Target path: ${TARGET_PATH}"
-        echo "Cache key: ${CACHE_KEY}"
-        echo "S3 bucket: s3://${S3_BUCKET}"
-        echo ""
-
-        # Normalize target path (expand tilde)
-        if [[ "${TARGET_PATH}" == ~* ]]; then
-          TARGET_PATH="${HOME}${TARGET_PATH:1}"
-        fi
-
-        # Canonicalize path (Linux only - macOS realpath doesn't support -m)
-        if [[ "$OSTYPE" == "linux-gnu"* ]]; then
-          TARGET_PATH=$(realpath -m "${TARGET_PATH}")
-        fi
-
-        echo "Normalized target path: ${TARGET_PATH}"
-        echo ""
-
-        # Debug: Show commit message
-        echo "=========================================="
-        echo "DEBUG: Cache clear tag detection"
-        echo "=========================================="
-        echo "Raw commit message:"
-        echo "${XAHAU_GA_COMMIT_MSG}"
-        echo ""
-        echo "Searching for: [ci-ga-clear-cache] or [ci-ga-clear-cache:*]"
-        echo ""
-
-        # Check for [ci-ga-clear-cache] tag in commit message (with optional search terms)
-        # Examples:
-        #   [ci-ga-clear-cache]              - Clear this job's cache
-        #   [ci-ga-clear-cache:ccache]       - Clear only if key contains "ccache"
-        #   [ci-ga-clear-cache:gcc Debug]    - Clear only if key contains both "gcc" AND "Debug"
-
-        # Extract search terms if present (e.g., "ccache" from "[ci-ga-clear-cache:ccache]")
-        SEARCH_TERMS=$(echo "${XAHAU_GA_COMMIT_MSG}" | grep -o '\[ci-ga-clear-cache:[^]]*\]' | sed 's/\[ci-ga-clear-cache://;s/\]//' || echo "")
-
-        SHOULD_CLEAR=false
-
-        if [ -n "${SEARCH_TERMS}" ]; then
-          # Search terms provided - check if THIS cache key matches ALL terms (AND logic)
-          echo "🔍 [ci-ga-clear-cache:${SEARCH_TERMS}] detected"
-          echo "Checking if cache key matches search terms..."
-          echo "  Cache key: ${CACHE_KEY}"
-          echo "  Search terms: ${SEARCH_TERMS}"
-          echo ""
-
-          MATCHES=true
-          for term in ${SEARCH_TERMS}; do
-            if ! echo "${CACHE_KEY}" | grep -q "${term}"; then
-              MATCHES=false
-              echo "  ✗ Key does not contain '${term}'"
-              break
-            else
-              echo "  ✓ Key contains '${term}'"
-            fi
-          done
-
-          if [ "${MATCHES}" = "true" ]; then
-            echo ""
-            echo "✅ Cache key matches all search terms - will clear cache"
-            SHOULD_CLEAR=true
-          else
-            echo ""
-            echo "⏭️  Cache key doesn't match search terms - skipping cache clear"
-          fi
-        elif echo "${XAHAU_GA_COMMIT_MSG}" | grep -q '\[ci-ga-clear-cache\]'; then
-          # No search terms - always clear this job's cache
-          echo "🗑️  [ci-ga-clear-cache] detected in commit message"
-          echo "Clearing cache for key: ${CACHE_KEY}"
-          SHOULD_CLEAR=true
-        fi
-
-        if [ "${SHOULD_CLEAR}" = "true" ]; then
-          echo ""
-
-          # Delete base layer
-          S3_BASE_KEY="s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst"
-          if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
-            echo "Deleting base layer: ${S3_BASE_KEY}"
-            aws s3 rm "${S3_BASE_KEY}" --region "${S3_REGION}" 2>/dev/null || true
-            echo "✓ Base layer deleted"
-          else
-            echo "ℹ️  No base layer found to delete"
-          fi
-
-          echo ""
-          echo "✅ Cache cleared successfully"
-          echo "Build will proceed from scratch (bootstrap mode)"
-          echo ""
-        fi
-
-        # Function to try restoring a cache key
-        try_restore_key() {
-          local key=$1
-          local s3_key="s3://${S3_BUCKET}/${key}-base.tar.zst"
-
-          echo "Checking for key: ${key}"
-
-          if aws s3 ls "${s3_key}" --region "${S3_REGION}" >/dev/null 2>&1; then
-            echo "✓ Found cache: ${s3_key}"
-            return 0
-          else
-            echo "✗ Not found: ${key}"
-            return 1
-          fi
-        }
-
-        # Try exact match first
-        MATCHED_KEY=""
-        EXACT_MATCH="false"
-
-        if try_restore_key "${CACHE_KEY}"; then
-          MATCHED_KEY="${CACHE_KEY}"
-          EXACT_MATCH="true"
-          echo ""
-          echo "🎯 Exact cache hit for key: ${CACHE_KEY}"
-        else
-          # Try restore-keys (prefix matching)
-          if [ -n "${RESTORE_KEYS}" ]; then
-            echo ""
-            echo "Primary key not found, trying restore-keys..."
-
-            while IFS= read -r restore_key; do
-              [ -z "${restore_key}" ] && continue
-              restore_key=$(echo "${restore_key}" | xargs)
-
-              if try_restore_key "${restore_key}"; then
-                MATCHED_KEY="${restore_key}"
-                EXACT_MATCH="false"
-                echo ""
-                echo "✓ Cache restored from fallback key: ${restore_key}"
-                break
-              fi
-            done <<< "${RESTORE_KEYS}"
-          fi
-        fi
-
-        # Check if we found anything
-        if [ -z "${MATCHED_KEY}" ]; then
-          echo ""
-          echo "❌ No cache found for key: ${CACHE_KEY}"
-
-          if [ "${FAIL_ON_MISS}" = "true" ]; then
-            echo "fail-on-cache-miss is enabled, failing workflow"
-            exit 1
-          fi
-
-          # Set outputs for cache miss
-          echo "cache-hit=false" >> $GITHUB_OUTPUT
-          echo "cache-primary-key=" >> $GITHUB_OUTPUT
-          echo "cache-matched-key=" >> $GITHUB_OUTPUT
-
-          # Create empty cache directory
-          mkdir -p "${TARGET_PATH}"
-
-          echo ""
-          echo "=========================================="
-          echo "Cache restore completed (bootstrap mode)"
-          echo "Created empty cache directory: ${TARGET_PATH}"
-          echo "=========================================="
-          exit 0
-        fi
-
-        # If lookup-only, we're done
-        if [ "${LOOKUP_ONLY}" = "true" ]; then
-          echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
-          echo "cache-primary-key=${CACHE_KEY}" >> $GITHUB_OUTPUT
-          echo "cache-matched-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
-
-          echo ""
-          echo "=========================================="
-          echo "Cache lookup completed (lookup-only mode)"
-          echo "Cache exists: ${MATCHED_KEY}"
-          echo "=========================================="
-          exit 0
-        fi
-
-        # Download and extract cache
-        S3_KEY="s3://${S3_BUCKET}/${MATCHED_KEY}-base.tar.zst"
-        TEMP_TARBALL="/tmp/xahau-cache-restore-$$.tar.zst"
-
-        echo ""
-        echo "Downloading cache..."
-        aws s3 cp "${S3_KEY}" "${TEMP_TARBALL}" --region "${S3_REGION}" --no-progress
-
-        TARBALL_SIZE=$(du -h "${TEMP_TARBALL}" | cut -f1)
-        echo "✓ Downloaded: ${TARBALL_SIZE}"
-
-        # Create parent directory if needed
-        mkdir -p "$(dirname "${TARGET_PATH}")"
-
-        # Remove existing target if it exists
-        if [ -e "${TARGET_PATH}" ]; then
-          echo "Removing existing target: ${TARGET_PATH}"
-          rm -rf "${TARGET_PATH}"
-        fi
-
-        # Create target directory and extract
-        mkdir -p "${TARGET_PATH}"
-        echo ""
-        echo "Extracting cache..."
-        zstd -d -c "${TEMP_TARBALL}" | tar -xf - -C "${TARGET_PATH}"
-        echo "✓ Cache extracted to: ${TARGET_PATH}"
-
-        # Cleanup
-        rm -f "${TEMP_TARBALL}"
-
-        # Set outputs
-        echo "cache-hit=${EXACT_MATCH}" >> $GITHUB_OUTPUT
-        echo "cache-primary-key=${CACHE_KEY}" >> $GITHUB_OUTPUT
-        echo "cache-matched-key=${MATCHED_KEY}" >> $GITHUB_OUTPUT
-
-        echo ""
-        echo "=========================================="
-        echo "Cache restore completed successfully"
-        echo "Cache hit: ${EXACT_MATCH}"
-        echo "Matched key: ${MATCHED_KEY}"
-        echo "=========================================="
--- a/.github/actions/xahau-ga-cache-save/action.yml
+++ b/.github/actions/xahau-ga-cache-save/action.yml
@@ -1,110 +0,0 @@
-name: 'Xahau Cache Save (S3)'
-description: 'Drop-in replacement for actions/cache/save using S3 storage'
-
-inputs:
-  path:
-    description: 'A list of files, directories, and wildcard patterns to cache (currently only single path supported)'
-    required: true
-  key:
-    description: 'An explicit key for saving the cache'
-    required: true
-  s3-bucket:
-    description: 'S3 bucket name for cache storage'
-    required: false
-    default: 'xahaud-github-actions-cache-niq'
-  s3-region:
-    description: 'S3 region'
-    required: false
-    default: 'us-east-1'
-  # Note: Composite actions can't access secrets.* directly - must be passed from workflow
-  aws-access-key-id:
-    description: 'AWS Access Key ID for S3 access'
-    required: true
-  aws-secret-access-key:
-    description: 'AWS Secret Access Key for S3 access'
-    required: true
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Save cache to S3
-      shell: bash
-      env:
-        AWS_ACCESS_KEY_ID: ${{ inputs.aws-access-key-id }}
-        AWS_SECRET_ACCESS_KEY: ${{ inputs.aws-secret-access-key }}
-        S3_BUCKET: ${{ inputs.s3-bucket }}
-        S3_REGION: ${{ inputs.s3-region }}
-        CACHE_KEY: ${{ inputs.key }}
-        TARGET_PATH: ${{ inputs.path }}
-      run: |
-        set -euo pipefail
-
-        echo "=========================================="
-        echo "Xahau Cache Save (S3)"
-        echo "=========================================="
-        echo "Target path: ${TARGET_PATH}"
-        echo "Cache key: ${CACHE_KEY}"
-        echo "S3 bucket: s3://${S3_BUCKET}"
-        echo ""
-
-        # Normalize target path (expand tilde and resolve to absolute path)
-        if [[ "${TARGET_PATH}" == ~* ]]; then
-          TARGET_PATH="${HOME}${TARGET_PATH:1}"
-        fi
-        echo "Normalized target path: ${TARGET_PATH}"
-        echo ""
-
-        # Check if target directory exists
-        if [ ! -d "${TARGET_PATH}" ]; then
-          echo "⚠️  Target directory does not exist: ${TARGET_PATH}"
-          echo "Skipping cache save."
-          exit 0
-        fi
-
-        # Use static base name (one base per key, immutable)
-        S3_BASE_KEY="s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst"
-
-        # Check if base already exists (immutability - first write wins)
-        if aws s3 ls "${S3_BASE_KEY}" --region "${S3_REGION}" >/dev/null 2>&1; then
-          echo "⚠️  Cache already exists: ${S3_BASE_KEY}"
-          echo "Skipping upload (immutability - first write wins, like GitHub Actions)"
-          echo ""
-          echo "=========================================="
-          echo "Cache save completed (already exists)"
-          echo "=========================================="
-          exit 0
-        fi
-
-        # Create tarball
-        BASE_TARBALL="/tmp/xahau-cache-base-$$.tar.zst"
-
-        echo "Creating cache tarball..."
-        tar -cf - -C "${TARGET_PATH}" . | zstd -3 -T0 -q -o "${BASE_TARBALL}"
-
-        BASE_SIZE=$(du -h "${BASE_TARBALL}" | cut -f1)
-        echo "✓ Cache tarball created: ${BASE_SIZE}"
-        echo ""
-
-        # Upload to S3
-        echo "Uploading cache to S3..."
-        echo "  Key: ${CACHE_KEY}-base.tar.zst"
-
-        aws s3api put-object \
-          --bucket "${S3_BUCKET}" \
-          --key "${CACHE_KEY}-base.tar.zst" \
-          --body "${BASE_TARBALL}" \
-          --tagging 'type=base' \
-          --region "${S3_REGION}" \
-          >/dev/null 2>&1
-
-        echo "✓ Uploaded: ${S3_BASE_KEY}"
-
-        # Cleanup
-        rm -f "${BASE_TARBALL}"
-
-        echo ""
-        echo "=========================================="
-        echo "Cache save completed successfully"
-        echo "Cache size: ${BASE_SIZE}"
-        echo "Cache key: ${CACHE_KEY}"
-        echo "=========================================="
--- a/.github/actions/xahau-ga-dependencies/action.yml
+++ b/.github/actions/xahau-ga-dependencies/action.yml
@@ -73,7 +73,7 @@ runs:
    - name: Restore Conan cache
      if: inputs.cache_enabled == 'true'
      id: cache-restore-conan
-      uses: ./.github/actions/xahau-ga-cache-restore
+      uses: ./.github/actions/xahau-actions-cache-restore
      with:
        path: ~/.conan2
        # Note: compiler-id format is compiler-version-stdlib[-gccversion]
@@ -81,6 +81,7 @@ runs:
        restore-keys: |
          ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}-
          ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-
+        use-deltas: 'false'
        aws-access-key-id: ${{ inputs.aws-access-key-id }}
        aws-secret-access-key: ${{ inputs.aws-secret-access-key }}

@@ -160,10 +161,11 @@ runs:
          ..

    - name: Save Conan cache
-      if: success() && inputs.cache_enabled == 'true' && steps.cache-restore-conan.outputs.cache-hit != 'true'
-      uses: ./.github/actions/xahau-ga-cache-save
+      if: always() && inputs.cache_enabled == 'true' && steps.cache-restore-conan.outputs.cache-hit != 'true'
+      uses: ./.github/actions/xahau-actions-cache-save
      with:
        path: ~/.conan2
        key: ${{ runner.os }}-conan-v${{ inputs.cache_version }}-${{ inputs.compiler-id }}-${{ hashFiles('**/conanfile.txt', '**/conanfile.py') }}-${{ inputs.configuration }}
+        use-deltas: 'false'
        aws-access-key-id: ${{ inputs.aws-access-key-id }}
        aws-secret-access-key: ${{ inputs.aws-secret-access-key }}
--- a/.github/actions/xahau-ga-get-commit-message/action.yml
+++ b/.github/actions/xahau-ga-get-commit-message/action.yml
@@ -1,73 +0,0 @@
-name: 'Get Commit Message'
-description: 'Gets commit message for both push and pull_request events and sets XAHAU_GA_COMMIT_MSG env var'
-
-inputs:
-  event-name:
-    description: 'The event name (push or pull_request)'
-    required: true
-  head-commit-message:
-    description: 'The head commit message (for push events)'
-    required: false
-    default: ''
-  pr-head-sha:
-    description: 'The PR head SHA (for pull_request events)'
-    required: false
-    default: ''
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Get commit message and set environment variable
-      shell: python
-      run: |
-        import os
-        import subprocess
-        import secrets
-
-        event_name = "${{ inputs.event-name }}"
-        pr_head_sha = "${{ inputs.pr-head-sha }}"
-
-        print("==========================================")
-        print("Setting XAHAU_GA_COMMIT_MSG environment variable")
-        print("==========================================")
-        print(f"Event: {event_name}")
-
-        if event_name == 'push':
-            # For push events, use the input directly
-            message = """${{ inputs.head-commit-message }}"""
-            print("Source: workflow input (github.event.head_commit.message)")
-        elif event_name == 'pull_request':
-            # For PR events, fetch the specific SHA
-            print(f"Source: git show {pr_head_sha} (fetching PR head commit)")
-
-            # Fetch the PR head commit
-            subprocess.run(
-                ['git', 'fetch', 'origin', pr_head_sha],
-                check=True
-            )
-
-            # Get commit message from the fetched SHA
-            result = subprocess.run(
-                ['git', 'show', '-s', '--format=%B', pr_head_sha],
-                capture_output=True,
-                text=True,
-                check=True
-            )
-            message = result.stdout.strip()
-        else:
-            message = ""
-            print(f"Warning: Unknown event type: {event_name}")
-
-        print(f"Commit message (first 100 chars): {message[:100]}")
-
-        # Write to GITHUB_ENV using heredoc with random delimiter (prevents injection attacks)
-        # See: https://securitylab.github.com/resources/github-actions-untrusted-input/
-        delimiter = f"EOF_{secrets.token_hex(16)}"
-
-        with open(os.environ['GITHUB_ENV'], 'a') as f:
-            f.write(f'XAHAU_GA_COMMIT_MSG<<{delimiter}\n')
-            f.write(message)
-            f.write(f'\n{delimiter}\n')
-
-        print(f"✓ XAHAU_GA_COMMIT_MSG set (available to all subsequent steps)")
-        print("==========================================")
--- a/.github/workflows/build-in-docker.yml.disabled
+++ b/.github/workflows/build-in-docker.yml.disabled
--- a/.github/workflows/clang-format.yml.disabled
+++ b/.github/workflows/clang-format.yml.disabled
--- a/.github/workflows/levelization.yml.disabled
+++ b/.github/workflows/levelization.yml.disabled
--- a/.github/workflows/test-cache-actions.yml.disabled
+++ b/.github/workflows/test-cache-actions.yml.disabled
@@ -0,0 +1,290 @@
+name: Test Cache Actions (State Machine)
+
+on:
+  push:
+    branches: ["nd-experiment-overlayfs-*"]
+  workflow_dispatch:
+    inputs:
+      state_assertion:
+        description: 'Expected state (optional, e.g. "2" to assert state 2)'
+        required: false
+        type: string
+        default: '1'
+      start_state:
+        description: 'Force specific starting state (optional, e.g. "3" to start at state 3)'
+        required: false
+        type: string
+      clear_cache:
+        description: 'Clear cache before running'
+        required: false
+        type: boolean
+        default: false
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test-cache-state-machine:
+    runs-on: ubuntu-latest
+    env:
+      CACHE_KEY: test-state-machine-${{ github.ref_name }}
+      CACHE_DIR: /tmp/test-cache
+      S3_BUCKET: xahaud-github-actions-cache-niq
+      S3_REGION: us-east-1
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Parse Inputs (workflow_dispatch or commit message)
+        id: parse-inputs
+        run: |
+          # Priority 1: workflow_dispatch inputs (manual trigger)
+          STATE_ASSERTION="${{ inputs.state_assertion }}"
+          START_STATE="${{ inputs.start_state }}"
+          SHOULD_CLEAR="${{ inputs.clear_cache }}"
+
+          # Priority 2: commit message tags (push event)
+          if [ "${{ github.event_name }}" = "push" ]; then
+            COMMIT_MSG="${{ github.event.head_commit.message }}"
+
+            # Parse [state:N] assertion tag (optional, if not provided as input)
+            if [ -z "${STATE_ASSERTION}" ] && echo "${COMMIT_MSG}" | grep -qE '\[state:[0-9]+\]'; then
+              STATE_ASSERTION=$(echo "${COMMIT_MSG}" | grep -oE '\[state:[0-9]+\]' | grep -oE '[0-9]+')
+              echo "State assertion found in commit: ${STATE_ASSERTION}"
+            fi
+
+            # Parse [start-state:N] force tag (optional, if not provided as input)
+            if [ -z "${START_STATE}" ] && echo "${COMMIT_MSG}" | grep -qE '\[start-state:[0-9]+\]'; then
+              START_STATE=$(echo "${COMMIT_MSG}" | grep -oE '\[start-state:[0-9]+\]' | grep -oE '[0-9]+')
+              echo "Start state found in commit: ${START_STATE}"
+            fi
+
+            # Parse [ci-clear-cache] tag (if not provided as input)
+            if [ "${SHOULD_CLEAR}" != "true" ] && echo "${COMMIT_MSG}" | grep -q '\[ci-clear-cache\]'; then
+              SHOULD_CLEAR=true
+              echo "Cache clear requested in commit"
+            fi
+          fi
+
+          # Output final values
+          echo "state_assertion=${STATE_ASSERTION}" >> "$GITHUB_OUTPUT"
+          echo "start_state=${START_STATE}" >> "$GITHUB_OUTPUT"
+          echo "should_clear=${SHOULD_CLEAR}" >> "$GITHUB_OUTPUT"
+
+          # Log what we're using
+          echo ""
+          echo "Configuration:"
+          [ -n "${STATE_ASSERTION}" ] && echo "  State assertion: ${STATE_ASSERTION}"
+          [ -n "${START_STATE}" ] && echo "  Start state: ${START_STATE}"
+          echo "  Clear cache: ${SHOULD_CLEAR}"
+
+      - name: Check S3 State (Before Restore)
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
+        run: |
+          echo "=========================================="
+          echo "S3 State Check (Before Restore)"
+          echo "=========================================="
+          echo "Cache key: ${CACHE_KEY}"
+          echo ""
+
+          # Check if base exists
+          BASE_EXISTS=false
+          if aws s3 ls "s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst" --region "${S3_REGION}" >/dev/null 2>&1; then
+            BASE_EXISTS=true
+          fi
+          echo "Base exists: ${BASE_EXISTS}"
+
+          # Count deltas
+          DELTA_COUNT=$(aws s3 ls "s3://${S3_BUCKET}/" --region "${S3_REGION}" | grep "${CACHE_KEY}-delta-" | wc -l || echo "0")
+          echo "Delta count: ${DELTA_COUNT}"
+
+      - name: Restore Cache
+        uses: ./.github/actions/xahau-actions-cache-restore
+        with:
+          path: ${{ env.CACHE_DIR }}
+          key: ${{ env.CACHE_KEY }}
+          s3-bucket: ${{ env.S3_BUCKET }}
+          s3-region: ${{ env.S3_REGION }}
+          use-deltas: 'true'
+          aws-access-key-id: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
+
+      - name: Auto-Detect State and Validate
+        id: state
+        env:
+          STATE_ASSERTION: ${{ steps.parse-inputs.outputs.state_assertion }}
+          START_STATE: ${{ steps.parse-inputs.outputs.start_state }}
+        run: |
+          echo "=========================================="
+          echo "State Detection and Validation"
+          echo "=========================================="
+
+          # Create cache directory if it doesn't exist
+          mkdir -p "${CACHE_DIR}"
+
+          # Handle [start-state:N] - force specific state
+          if [ -n "${START_STATE}" ]; then
+            echo "🎯 [start-state:${START_STATE}] detected - forcing state setup"
+
+            # Clear cache and create state files 0 through START_STATE
+            rm -f ${CACHE_DIR}/state*.txt 2>/dev/null || true
+            for i in $(seq 0 ${START_STATE}); do
+              echo "State ${i} - Forced at $(date)" > "${CACHE_DIR}/state${i}.txt"
+              echo "Commit: ${{ github.sha }}" >> "${CACHE_DIR}/state${i}.txt"
+            done
+
+            DETECTED_STATE=${START_STATE}
+            echo "✓ Forced to state ${DETECTED_STATE}"
+          else
+            # Auto-detect state by counting state files
+            STATE_FILES=$(ls ${CACHE_DIR}/state*.txt 2>/dev/null | wc -l)
+            DETECTED_STATE=${STATE_FILES}
+            echo "Auto-detected state: ${DETECTED_STATE} (${STATE_FILES} state files)"
+          fi
+
+          # Show cache contents
+          echo ""
+          echo "Cache contents:"
+          if [ -d "${CACHE_DIR}" ] && [ "$(ls -A ${CACHE_DIR})" ]; then
+            ls -la "${CACHE_DIR}"
+          else
+            echo "(empty)"
+          fi
+
+          # Validate [state:N] assertion if provided
+          if [ -n "${STATE_ASSERTION}" ]; then
+            echo ""
+            echo "Validating assertion: [state:${STATE_ASSERTION}]"
+            if [ "${DETECTED_STATE}" -ne "${STATE_ASSERTION}" ]; then
+              echo "❌ ERROR: State mismatch!"
+              echo "   Expected (from [state:N]): ${STATE_ASSERTION}"
+              echo "   Detected (from cache):     ${DETECTED_STATE}"
+              exit 1
+            fi
+            echo "✓ Assertion passed: detected == expected (${DETECTED_STATE})"
+          fi
+
+          # Output detected state for next steps
+          echo "detected_state=${DETECTED_STATE}" >> "$GITHUB_OUTPUT"
+
+          echo ""
+          echo "=========================================="
+
+      - name: Simulate Build (State Transition)
+        env:
+          DETECTED_STATE: ${{ steps.state.outputs.detected_state }}
+        run: |
+          echo "=========================================="
+          echo "Simulating Build (State Transition)"
+          echo "=========================================="
+
+          # Calculate next state
+          NEXT_STATE=$((DETECTED_STATE + 1))
+          echo "Transitioning: State ${DETECTED_STATE} → State ${NEXT_STATE}"
+          echo ""
+
+          # Create state file for next state
+          STATE_FILE="${CACHE_DIR}/state${NEXT_STATE}.txt"
+          echo "State ${NEXT_STATE} - Created at $(date)" > "${STATE_FILE}"
+          echo "Commit: ${{ github.sha }}" >> "${STATE_FILE}"
+          echo "Message: ${{ github.event.head_commit.message }}" >> "${STATE_FILE}"
+
+          echo "✓ Created ${STATE_FILE}"
+
+          # Show final cache state
+          echo ""
+          echo "Final cache contents:"
+          ls -la "${CACHE_DIR}"
+
+          echo ""
+          echo "State files:"
+          cat ${CACHE_DIR}/state*.txt
+
+      - name: Save Cache
+        uses: ./.github/actions/xahau-actions-cache-save
+        with:
+          path: ${{ env.CACHE_DIR }}
+          key: ${{ env.CACHE_KEY }}
+          s3-bucket: ${{ env.S3_BUCKET }}
+          s3-region: ${{ env.S3_REGION }}
+          use-deltas: 'true'
+          aws-access-key-id: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
+
+      - name: Validate S3 State (After Save)
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
+          DETECTED_STATE: ${{ steps.state.outputs.detected_state }}
+        run: |
+          echo "=========================================="
+          echo "S3 State Validation (After Save)"
+          echo "=========================================="
+
+          # Calculate next state (what we just saved)
+          NEXT_STATE=$((DETECTED_STATE + 1))
+          echo "Saved state: ${NEXT_STATE}"
+          echo ""
+
+          # Check if base exists
+          if aws s3 ls "s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst" --region "${S3_REGION}" >/dev/null 2>&1; then
+            BASE_SIZE=$(aws s3 ls "s3://${S3_BUCKET}/${CACHE_KEY}-base.tar.zst" --region "${S3_REGION}" | awk '{print $3}')
+            echo "✓ Base exists: ${CACHE_KEY}-base.tar.zst (${BASE_SIZE} bytes)"
+          else
+            echo "❌ ERROR: Base should exist after save"
+            exit 1
+          fi
+
+          # List deltas
+          echo ""
+          echo "Delta layers:"
+          DELTAS=$(aws s3 ls "s3://${S3_BUCKET}/" --region "${S3_REGION}" | grep "${CACHE_KEY}-delta-" || echo "")
+          if [ -n "${DELTAS}" ]; then
+            echo "${DELTAS}"
+            DELTA_COUNT=$(echo "${DELTAS}" | wc -l)
+          else
+            echo "(none)"
+            DELTA_COUNT=0
+          fi
+
+          # Validate S3 state
+          echo ""
+          if [ "${DETECTED_STATE}" -eq 0 ]; then
+            # Saved state 1 from bootstrap (state 0 → 1)
+            if [ "${DELTA_COUNT}" -ne 0 ]; then
+              echo "⚠️  WARNING: Bootstrap (state 1) should have 0 deltas, found ${DELTA_COUNT}"
+            else
+              echo "✓ State 1 saved: base exists, 0 deltas"
+            fi
+          else
+            # Saved delta (state N+1)
+            if [ "${DELTA_COUNT}" -ne 1 ]; then
+              echo "⚠️  WARNING: State ${NEXT_STATE} expects 1 delta (inline cleanup), found ${DELTA_COUNT}"
+              echo "This might be OK if multiple builds ran concurrently"
+            else
+              echo "✓ State ${NEXT_STATE} saved: base + 1 delta (old deltas cleaned)"
+            fi
+          fi
+
+          echo ""
+          echo "=========================================="
+          echo "✅ State ${DETECTED_STATE} → ${NEXT_STATE} Complete!"
+          echo "=========================================="
+          echo ""
+          echo "Next commit will auto-detect state ${NEXT_STATE}"
+          echo ""
+          echo "Options:"
+          echo "  # Normal (auto-advance)"
+          echo "  git commit -m 'continue testing'"
+          echo ""
+          echo "  # With assertion (validate state)"
+          echo "  git commit -m 'test delta [state:${NEXT_STATE}]'"
+          echo ""
+          echo "  # Clear cache and restart"
+          echo "  git commit -m 'fresh start [ci-clear-cache]'"
+          echo ""
+          echo "  # Jump to specific state"
+          echo "  git commit -m 'jump to state 3 [start-state:3]'"
--- a/.github/workflows/test-overlayfs-delta.yml.disabled
+++ b/.github/workflows/test-overlayfs-delta.yml.disabled
@@ -0,0 +1,182 @@
+name: Test OverlayFS Delta Extraction
+
+on:
+  push:
+    branches: ["*"]
+  workflow_dispatch:
+
+jobs:
+  test-overlayfs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      # - name: Test encrypted secrets (decrypt test message)
+      #   run: |
+      #     echo "========================================"
+      #     echo "TESTING ENCRYPTED SECRETS"
+      #     echo "========================================"
+      #     echo ""
+      #     echo "Decrypting test message from .github/secrets/test-message.gpg"
+      #     echo "Using encryption key from GitHub Secrets..."
+      #     echo ""
+      #
+      #     # Decrypt using key from GitHub Secrets
+      #     echo "${{ secrets.TEST_ENCRYPTION_KEY }}" | \
+      #       gpg --batch --yes --passphrase-fd 0 \
+      #       --decrypt .github/secrets/test-message.gpg
+      #
+      #     echo ""
+      #     echo "========================================"
+      #     echo "If you see the success message above,"
+      #     echo "then encrypted secrets work! 🎉"
+      #     echo "========================================"
+      #     echo ""
+
+      - name: Setup OverlayFS layers
+        run: |
+          echo "=== Creating directory structure ==="
+          mkdir -p /tmp/test/{base,delta,upper,work,merged}
+
+          echo "=== Creating base layer files ==="
+          echo "base file 1" > /tmp/test/base/file1.txt
+          echo "base file 2" > /tmp/test/base/file2.txt
+          echo "base file 3" > /tmp/test/base/file3.txt
+          mkdir -p /tmp/test/base/subdir
+          echo "base subdir file" > /tmp/test/base/subdir/file.txt
+
+          echo "=== Base layer contents ==="
+          find /tmp/test/base -type f -exec sh -c 'echo "{}:"; cat "{}"' \;
+
+          echo "=== Mounting OverlayFS ==="
+          sudo mount -t overlay overlay \
+            -o lowerdir=/tmp/test/base,upperdir=/tmp/test/upper,workdir=/tmp/test/work \
+            /tmp/test/merged
+
+          echo "=== Mounted successfully ==="
+          mount | grep overlay
+
+      - name: Verify merged view shows base files
+        run: |
+          echo "=== Contents of /merged (should show base files) ==="
+          ls -R /tmp/test/merged
+          find /tmp/test/merged -type f -exec sh -c 'echo "{}:"; cat "{}"' \;
+
+      - name: Make changes via merged layer
+        run: |
+          echo "=== Making changes via /merged ==="
+
+          # Overwrite existing file
+          echo "MODIFIED file 2" > /tmp/test/merged/file2.txt
+          echo "Modified file2.txt"
+
+          # Create new file
+          echo "NEW file 4" > /tmp/test/merged/file4.txt
+          echo "Created new file4.txt"
+
+          # Create new directory with file
+          mkdir -p /tmp/test/merged/newdir
+          echo "NEW file in new dir" > /tmp/test/merged/newdir/newfile.txt
+          echo "Created newdir/newfile.txt"
+
+          # Add file to existing directory
+          echo "NEW file in existing subdir" > /tmp/test/merged/subdir/newfile.txt
+          echo "Created subdir/newfile.txt"
+
+          echo "=== Changes complete ==="
+
+      - name: Show the delta (upperdir)
+        run: |
+          echo "========================================"
+          echo "THE DELTA (only changes in /upper):"
+          echo "========================================"
+
+          if [ -z "$(ls -A /tmp/test/upper)" ]; then
+            echo "Upper directory is empty - no changes detected"
+          else
+            echo "Upper directory structure:"
+            ls -R /tmp/test/upper
+            echo ""
+            echo "Upper directory files with content:"
+            find /tmp/test/upper -type f -exec sh -c 'echo "---"; echo "FILE: {}"; cat "{}"; echo ""' \;
+
+            echo "========================================"
+            echo "SIZE OF DELTA:"
+            du -sh /tmp/test/upper
+            echo "========================================"
+          fi
+
+      - name: Compare base vs upper vs merged
+        run: |
+          echo "========================================"
+          echo "COMPARISON:"
+          echo "========================================"
+
+          echo "BASE layer (original, untouched):"
+          ls -la /tmp/test/base/
+          echo ""
+
+          echo "UPPER layer (DELTA - only changes):"
+          ls -la /tmp/test/upper/
+          echo ""
+
+          echo "MERGED layer (unified view = base + upper):"
+          ls -la /tmp/test/merged/
+          echo ""
+
+          echo "========================================"
+          echo "PROOF: Upper dir contains ONLY the delta!"
+          echo "========================================"
+
+      - name: Simulate tarball creation (what we'd upload)
+        run: |
+          echo "=== Creating tarball of delta ==="
+          tar -czf /tmp/delta.tar.gz -C /tmp/test/upper .
+
+          echo "Delta tarball size:"
+          ls -lh /tmp/delta.tar.gz
+
+          echo ""
+          echo "Delta tarball contents:"
+          tar -tzf /tmp/delta.tar.gz
+
+          echo ""
+          echo "========================================"
+          echo "This is what we'd upload to S3/rsync!"
+          echo "Only ~few KB instead of entire cache!"
+          echo "========================================"
+
+      - name: Upload delta to S3 (actual test!)
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}
+        run: |
+          echo "========================================"
+          echo "UPLOADING TO S3"
+          echo "========================================"
+
+          # Upload the delta tarball
+          aws s3 cp /tmp/delta.tar.gz \
+            s3://xahaud-github-actions-cache-niq/hello-world-first-test.tar.gz \
+            --region us-east-1
+
+          echo ""
+          echo "✅ Successfully uploaded to S3!"
+          echo "File: s3://xahaud-github-actions-cache-niq/hello-world-first-test.tar.gz"
+          echo ""
+
+          # Verify it exists
+          echo "Verifying upload..."
+          aws s3 ls s3://xahaud-github-actions-cache-niq/hello-world-first-test.tar.gz --region us-east-1
+
+          echo ""
+          echo "========================================"
+          echo "S3 upload test complete! 🚀"
+          echo "========================================"
+
+      - name: Cleanup
+        if: always()
+        run: |
+          echo "=== Unmounting OverlayFS ==="
+          sudo umount /tmp/test/merged || true
--- a/.github/workflows/verify-generated-headers.yml.disabled
+++ b/.github/workflows/verify-generated-headers.yml.disabled
--- a/.github/workflows/xahau-ga-macos.yml.disabled
+++ b/.github/workflows/xahau-ga-macos.yml.disabled
@@ -30,14 +30,6 @@ jobs:
      - name: Checkout
        uses: actions/checkout@v4

-      - name: Get commit message
-        id: get-commit-message
-        uses: ./.github/actions/xahau-ga-get-commit-message
-        with:
-          event-name: ${{ github.event_name }}
-          head-commit-message: ${{ github.event.head_commit.message }}
-          pr-head-sha: ${{ github.event.pull_request.head.sha }}
-
      - name: Install Conan
        run: |
          brew install conan
@@ -86,6 +78,14 @@ jobs:
      - name: Install ccache
        run: brew install ccache

+      - name: Configure ccache
+        uses: ./.github/actions/xahau-configure-ccache
+        with:
+          max_size: 2G
+          hash_dir: true
+          compiler_check: content
+          is_main_branch: ${{ github.ref_name == env.MAIN_BRANCH_NAME }}
+
      - name: Check environment
        run: |
          echo "PATH:"
@@ -130,9 +130,6 @@ jobs:
          compiler-id: clang
          cache_version: ${{ env.CACHE_VERSION }}
          main_branch: ${{ env.MAIN_BRANCH_NAME }}
-          stdlib: libcxx
-          aws-access-key-id: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_AWS_ACCESS_KEY }}

      - name: Test
        run: |
--- a/.github/workflows/xahau-ga-nix.yml
+++ b/.github/workflows/xahau-ga-nix.yml
@@ -162,14 +162,6 @@ jobs:
      - name: Checkout
        uses: actions/checkout@v4

-      - name: Get commit message
-        id: get-commit-message
-        uses: ./.github/actions/xahau-ga-get-commit-message
-        with:
-          event-name: ${{ github.event_name }}
-          head-commit-message: ${{ github.event.head_commit.message }}
-          pr-head-sha: ${{ github.event.pull_request.head.sha }}
-
      - name: Install build dependencies
        run: |
          sudo apt-get update
@@ -239,6 +231,13 @@ jobs:
          # Install Conan 2
          pip install --upgrade "conan>=2.0,<3"

+      - name: Configure ccache
+        uses: ./.github/actions/xahau-configure-ccache
+        with:
+          max_size: 2G
+          hash_dir: true
+          compiler_check: content
+
      - name: Check environment
        run: |
          echo "PATH:"
--- a/Builds/levelization/README.md
+++ b/Builds/levelization/README.md
@@ -72,15 +72,15 @@ It generates many files of [results](results):
  desired as described above. In a perfect repo, this file will be
  empty.
  This file is committed to the repo, and is used by the [levelization
-  Github workflow](../../.github/workflows/levelization.yml) to validate
+  Github workflow](../../.github/workflows/levelization.yml.disabled) to validate
  that nothing changed.
 * [`ordering.txt`](results/ordering.txt): A list showing relationships
  between modules where there are no loops as they actually exist, as
  opposed to how they are desired as described above.
  This file is committed to the repo, and is used by the [levelization
-  Github workflow](../../.github/workflows/levelization.yml) to validate
+  Github workflow](../../.github/workflows/levelization.yml.disabled) to validate
  that nothing changed.
-* [`levelization.yml`](../../.github/workflows/levelization.yml)
+* [`levelization.yml`](../../.github/workflows/levelization.yml.disabled)
  Github Actions workflow to test that levelization loops haven't
  changed.  Unfortunately, if changes are detected, it can't tell if
  they are improvements or not, so if you have resolved any issues or
Author	SHA1	Message	Date
Nicholas Dudfield	a4f96a435a	fix: don't override ccache's default cache_dir PROBLEM: Setting cache_dir='~/.ccache' explicitly stores the LITERAL tilde in ccache.conf, which might not expand properly. SOLUTION: Don't set cache_dir at all - let ccache use its default - Linux default: ~/.ccache - ccache will expand the tilde correctly when using default - Only configure max_size, hash_dir, compiler_check - Remove CCACHE_DIR env var (not needed with default) This should fix the issue where ccache runs but creates no cache files.	2025-10-30 17:58:08 +07:00
Nicholas Dudfield	d0f63cc2d1	debug: add ccache directory inspection after build Add comprehensive debug output to inspect ~/.ccache contents: - Full recursive directory listing - Disk space check (rule out disk full) - ccache config verification - Directory sizes - List actual files (distinguish config from cache data) This will help diagnose why ccache is being invoked on every compilation but not creating any cache files (0.0 GB).	2025-10-30 17:53:33 +07:00
Nicholas Dudfield	2433bfe277	debug: add ninja verbose output to see compile commands Temporarily adding -v flag to ninja builds to see actual compile commands. This will show whether ccache is actually being invoked or if the CMAKE_C_COMPILER_LAUNCHER setting isn't taking effect. We should see either: ccache /usr/bin/g++-13 -c file.cpp (working) /usr/bin/g++-13 -c file.cpp (not working) This is temporary for debugging - remove once ccache issue resolved.	2025-10-30 17:04:00 +07:00
Nicholas Dudfield	ef40a7f351	refactor: move Conan profile creation after cache restore PROBLEM: Profile was created before cache restore, then immediately overwritten by cached profile. This meant we were using a potentially stale cached profile instead of fresh configuration. SOLUTION: Move profile creation into dependencies action after restore - Dependencies action now takes compiler params as inputs - Profile created AFTER cache restore (overwrites any cached profile) - Ensures fresh profile with correct compiler settings for each job CHANGES: - Dependencies action: Add inputs (os, arch, compiler, compiler_version, cc, cxx) - Dependencies action: Add 'Configure Conan' step after cache restore - Dependencies action: Support both Linux and macOS profile generation - Nix workflow: Remove 'Configure Conan' step, pass compiler params - macOS workflow: Detect compiler version, pass to dependencies action Benefits: ✅ Profile always fresh and correct for current matrix config ✅ Cache still includes .conan.db and other important files ✅ Self-contained dependencies action (easier to understand) ✅ Works for both Linux (with explicit cc/cxx) and macOS (auto-detect)	2025-10-30 16:41:43 +07:00
Nicholas Dudfield	a4a4126bdc	fix: indent heredoc content for valid YAML syntax The heredoc content inside the run block needs proper indentation to be valid YAML. Without indentation, the YAML parser treats the lines as top-level keys, causing syntax errors. The generated CMake file will have leading spaces, but CMake handles this fine (whitespace-tolerant parser).	2025-10-30 16:17:40 +07:00
Nicholas Dudfield	0559b6c418	fix: enable ccache for main app via wrapper toolchain PROBLEM: ccache was never being used (all builds compiling from scratch) - CMake command-line args (-DCMAKE_C_COMPILER_LAUNCHER=ccache) were being overridden by Conan's conan_toolchain.cmake - This toolchain file loads AFTER command-line args and resets settings - Result: Empty ccache entries (~200 bytes) in both old and new cache SOLUTION: Wrapper toolchain that overlays ccache on Conan's toolchain - Create wrapper_toolchain.cmake that includes Conan's toolchain first - Then sets CMAKE_C_COMPILER_LAUNCHER and CMAKE_CXX_COMPILER_LAUNCHER - This happens AFTER Conan's toolchain loads, so settings stick - Only affects main app build (NOT Conan dependency builds) CHANGES: - Build action: Create wrapper toolchain when ccache enabled - Build action: Use CMAKE_CURRENT_LIST_DIR for correct relative path - Build action: Remove broken CCACHE_ARGS logic (was being overridden) - Build action: Use ${TOOLCHAIN_FILE} variable instead of hardcoded path This approach: ✅ Conan dependency builds: Clean (no ccache overhead) ✅ Main xahaud build: Uses ccache via wrapper ✅ Separation: Build action controls ccache, not Conan profile ✅ Simple: No profile regeneration, just one wrapper file	2025-10-30 16:06:45 +07:00
Nicholas Dudfield	f8d1a6f2b4	fix: normalize paths in cache actions to fix bootstrap detection Path comparison was failing when registry had expanded paths (/home/runner/.ccache) but input had unexpanded paths (~/.ccache), causing bootstrap mode to not be detected. Now both restore and save actions consistently expand tildes to absolute paths before writing to or reading from the mount registry.	2025-10-30 13:25:55 +07:00
Nicholas Dudfield	c46ede7c8f	chore: bump CACHE_VERSION to 3 for fresh bootstrap Triggering clean rebuild after S3 cache cleared and ccache simplified to single directory.	2025-10-30 11:57:02 +07:00
Nicholas Dudfield	0e2bc365ea	refactor: simplify ccache to single directory with restore-keys fallback Removed split ccache configuration (~/.ccache-main + ~/.ccache-current). The split had an edge case: building a feature branch before main branch cache is primed results in double ccache disk usage: - Feature branch builds first → populates ~/.ccache-current (2G max_size) - Main branch builds later → populates ~/.ccache-main (2G max_size) - Next feature branch build → restores both caches (4G total) This doubles ccache disk usage unnecessarily on GitHub-hosted runners. New single-directory approach: - Single ~/.ccache directory for all branches - Branch-specific keys with restore-keys fallback to main branch - Cache actions handle cross-branch sharing via partial-match mode - Simpler and prevents double disk usage edge case Changes: - xahau-configure-ccache: Single cache_dir input, removed branch logic - xahau-ga-build: Single restore/save step, uses restore-keys for fallback - xahau-ga-nix: Removed is_main_branch parameter Disk usage: 2G max (vs 4G worst case with split)	2025-10-30 11:24:14 +07:00
Nicholas Dudfield	446bc76b69	fix: remove broken cache save conditions for Conan The check-conanfile-changes logic prevented cache saves on feature branches unless conanfile changed, but this was broken because: 1. Cache keys don't include branch name (shared across branches) 2. S3 immutability makes this unnecessary (first-write-wins) 3. It prevented bootstrap saves on feature branches Also removed unused safe-branch step (dead code from copy-paste)	2025-10-30 10:35:37 +07:00
Nicholas Dudfield	a0c38a4fb3	fix: disable deltas for Conan cache (base-only mode) Conan dependencies are relatively static, so use base-only mode instead of delta caching to avoid unnecessary complexity	2025-10-30 10:03:48 +07:00
Nicholas Dudfield	631650f7eb	feat(wip): add nd-experiment-overlayfs-2025-10-29 to nix push	2025-10-30 09:56:33 +07:00
Nicholas Dudfield	0b31d8e534	feat: replace actions/cache with custom S3+OverlayFS cache - Updated xahau-ga-dependencies to use xahau-actions-cache-restore/save - Updated xahau-ga-build to use custom cache for ccache (4 operations) - Added AWS credential inputs to both actions - Main workflow now passes AWS secrets to cache actions - Removed legacy ~/.conan path (Conan 2 uses ~/.conan2 only) - All cache keys remain unchanged for compatibility	2025-10-30 09:55:06 +07:00
Nicholas Dudfield	ecf03f4afe	test: expect state 1 [ci-clear-cache]	2025-10-30 09:50:57 +07:00
Nicholas Dudfield	b801c2837d	test: expect state 3	2025-10-30 09:46:17 +07:00
Nicholas Dudfield	1474e808cb	test: expect state 2	2025-10-30 09:43:26 +07:00
Nicholas Dudfield	457e633a81	feat(test): add workflow_dispatch inputs for state machine testing Instead of requiring commits with message tags, allow manual workflow triggering with inputs: - state_assertion: Expected state for validation - start_state: Force specific starting state - clear_cache: Clear cache before running Benefits: - No need for empty commits to trigger tests - Easy manual testing via GitHub Actions UI - Still supports commit message tags for push events - Workflow inputs take priority over commit tags	2025-10-30 09:40:26 +07:00
Nicholas Dudfield	7ea99caa19	fix(cache): trim whitespace in delta count + use s3api for tagging Two fixes: 1. Restore: Trim whitespace from DELTA_COUNT to fix integer comparison 2. Save: Use 's3api put-object' instead of 's3 cp' to support --tagging The aws s3 cp command doesn't support --tagging parameter. Switched to s3api put-object which supports tagging directly. Tags are needed for lifecycle policies (30-day eviction).	2025-10-30 09:29:02 +07:00
Nicholas Dudfield	3e5c15c172	fix(cache): handle empty cache gracefully in [ci-clear-cache] When [ci-clear-cache] is used but no cache exists yet, grep returns exit code 1 which causes script failure with set -e. Add \|\| echo "0" to handle case where no deltas exist to delete.	2025-10-30 09:25:21 +07:00
Nicholas Dudfield	52b4fb503c	feat(cache): implement [ci-clear-cache] tag + auto-detecting state machine test Cache Clearing Feature: - Add [ci-clear-cache] commit message tag detection in restore action - Deletes base + all deltas when tag present - Implicit access via github.event.head_commit.message env var - No workflow changes needed (action handles automatically) - Commit message only (not PR title) - one-time action State Machine Test Workflow: - Auto-detects state by counting state files (state0.txt, state1.txt, etc.) - Optional [state:N] assertions validate detected == expected - [start-state:N] forces specific state for scenario testing - Dual validation: local cache state AND S3 objects - 4 validation checkpoints: S3 before, local after restore, after build, S3 after save - Self-documenting: prints next steps after each run - Supports [ci-clear-cache] integration Usage: # Auto-advance (normal) git commit -m 'continue testing' # With assertion git commit -m 'test delta [state:2]' # Clear and restart git commit -m 'fresh start [ci-clear-cache]' # Jump to scenario git commit -m 'test from state 3 [start-state:3]'	2025-10-30 09:19:12 +07:00
Nicholas Dudfield	98123fa934	feat(cache): implement inline delta cleanup (keep only 1 per key) - Add automatic cleanup after each delta upload - Query all deltas for key, sort by LastModified - Keep only latest (just uploaded), delete all older ones - Matches restore logic (only uses latest delta) - Minimal storage: 1 delta per key (~2GB) vs unbounded growth - Simpler than keeping N: restore never needs older deltas - Concurrency-safe (idempotent batch deletes) - Eliminates need for separate cleanup workflow Rationale: Since restore only ever uses the single latest delta, keeping historical deltas adds complexity without benefit. This matches GitHub Actions semantics (one 'latest' per key).	2025-10-29 14:51:28 +07:00
Nicholas Dudfield	ce7b1c4f1d	feat: add custom S3+OverlayFS cache actions with configurable delta support Implements drop-in replacement for actions/cache using S3 backend and OverlayFS for delta caching: - xahau-actions-cache-restore: Downloads immutable base + optional latest delta - xahau-actions-cache-save: Saves immutable bases (bootstrap/partial-match) or timestamped deltas (exact-match) Key features: - Immutable bases: One static base per key (first-write-wins, GitHub Actions semantics) - Timestamped deltas: Always-timestamped to eliminate concurrency issues - Configurable use-deltas parameter (default true): - true: For symbolic keys (branch-based) - massive bandwidth savings via incremental deltas - false: For content-based keys (hash-based) - base-only mode, no delta complexity - Three cache modes: bootstrap, partial-match (restore-keys), exact-match - OverlayFS integration: Automatic delta extraction via upperdir, whiteout file support - S3 lifecycle ready: Bases tagged 'type=base', deltas tagged 'type=delta-archive' Decision rule for use-deltas: - Content-based discriminator (hashFiles, commit SHA) → use-deltas: false - Symbolic discriminator (branch name, tag, PR) → use-deltas: true Also disables existing workflows temporarily during development.	2025-10-29 13:07:40 +07:00
Nicholas Dudfield	e062dcae58	feat(wip): comment out unused secret encryption	2025-10-29 09:05:17 +07:00
Nicholas Dudfield	a9d284fec1	feat(wip): use new key names	2025-10-29 09:02:24 +07:00
Nicholas Dudfield	065d0c3e07	feat(wip): remove currently unused workflows	2025-10-29 08:57:21 +07:00
Nicholas Dudfield	4fda40b709	test: add S3 upload to overlayfs delta test - Upload delta tarball to S3 bucket - Test file: hello-world-first-test.tar.gz - Uses new secret names: XAHAUD_GITHUB_ACTIONS_CACHE_NIQ_KEY_ID/ACCESS_KEY - Verifies upload with aws s3 ls - Complete end-to-end test: OverlayFS → tarball → S3	2025-10-29 08:49:05 +07:00
Nicholas Dudfield	6014356d91	test: add encrypted secrets test to overlayfs workflow - Generate random encryption key and store in GitHub Secrets via gh CLI - Encrypt test message with GPG and commit to repo - Decrypt in workflow using key from secrets and echo result - Demonstrates encrypted secrets approach for SSH keys	2025-10-29 08:04:28 +07:00
Nicholas Dudfield	d790f97430	feat(wip): experiment overlayfs	2025-10-29 07:52:06 +07:00