Mastering the debug-action-cache: A Guide to Troubleshooting Remote Build Execution
In the world of modern software engineering, speed is everything. Tools like Bazel, Buck, and various CI/CD runners leverage Action Caching to ensure that if a task has been done once, it never has to be done again. But what happens when your cache returns a "hit" that is actually broken, or a "miss" that should have been a hit?
Enter the debug-action-cache. Whether you are working with Bazel’s remote execution flags or custom CI scripts, debugging the action cache is the "black belt" skill of build engineering. What is the Action Cache?
Before diving into debugging, let’s define the beast. The Action Cache (AC) stores the mapping between an Action Digest (a hash of your command line, environment variables, and input file metadata) and the Action Result (the logs, exit codes, and output files).
A "Cache Miss" occurs when the digest doesn't match anything in the database, forcing a re-run. A "Cache Hit" occurs when the digest matches, allowing the system to simply download the previous results. Why Debug the Cache?
You usually start looking into debug-action-cache for two reasons:
Non-Determinism (The Poisoned Cache): Two different machines run the "same" code but produce different outputs, or a build passes locally but fails remotely.
Cache Misses (The Slow Build): You’ve changed nothing, yet the build is recompiling everything from scratch. Key Strategies for Debugging 1. Inspecting the Input Root
The most common cause of a cache miss is a "dirty" input root. Even a timestamp change or a stray .git directory file can change the Action Digest.
The Tooling: Use flags like --execution_log_json_file (in Bazel) to dump exactly what was sent to the cache.
The Fix: Compare two logs using a diff tool. Look for absolute paths that should be relative, or environment variables like PATH or PWD that differ between developer machines. 2. Analyzing the "Action Key" debug-action-cache
The Action Key is the fingerprint of the work. If you are using a tool that supports a debug-action-cache flag, it will often output the raw components used to generate this key. Check for: Command Line Arguments: Did a compiler flag move?
Environment Variables: Is USER or HOSTNAME being sucked into the action?
Platform Properties: Does the remote worker have a different OS version or CPU architecture than the previous run? 3. Content Addressable Storage (CAS) Validation
Sometimes the cache hit is correct, but the files it points to are corrupted or missing from the CAS.
Debug Step: Use a tool to manually query the Digest. If the Action Cache returns a result, but the ActionResult references a file hash that doesn't exist in the CAS, your cache is "orphaned." Common Culprits of Cache Issues
Absolute Paths: Compilers often embed absolute paths into .d or .o files. If the build directory isn't identical across machines, the cache will miss or provide "poisoned" binaries containing the wrong paths.
Non-Deterministic Tools: Some code generators include the current date/time in the header of the generated file. This invalidates the cache every single minute.
Missing Inputs: If an input isn't explicitly declared in the build graph but is read by the tool (like a global config file), the cache won't know to invalidate when that file changes. Best Practices for a Healthy Cache
Hermeticity: Ensure your build actions don't access the internet or arbitrary file system locations.
Canonicalize Paths: Always use relative paths for compiler inputs. If you see a segmentation error (e
Strip Metadata: Use flags to prevent compilers from adding timestamps to output binaries.
Log Everything: Keep "Before" and "After" execution logs whenever you make changes to your build infrastructure. Conclusion
Debugging the action cache feels like detective work. By using debug-action-cache methodologies to peel back the layers of digests and fingerprints, you move from "guessing why the build is slow" to "knowing exactly why it’s re-running."
In the race for sub-minute build times, a transparent and debuggable cache is your most powerful asset.
Are you seeing specific cache misses in a particular build system like Bazel or GitHub Actions, or
This guide provides a comprehensive overview of debugging the GitHub Actions cache
, a common area of frustration when build times spike due to unexpected cache misses or corrupted data. 1. Enable Verbose Debug Logging
Before digging into specific keys, you need to see exactly what the runner is doing during the "Restore Cache" and "Post-run: Save Cache" steps. Repository Variable Settings > Secrets and variables > Actions ACTIONS_STEP_DEBUG Workflow Scope : This will force the actions/cache
step to output detailed logs, including the internal API calls used to check for existing cache versions. GitHub Docs 2. Inspecting Cache States via the UI
provides a dedicated management interface to view and delete active caches. Navigation : Navigate to your repository on GitHub. Go to (left sidebar under "Management"). What to Look For Common Problems & Solutions | Symptom | Likely
: Caches have a 10GB limit per repository. If you hit this, GitHub will evict older caches, which might explain sudden "misses" for older branches.
: Identify "stale" caches that haven't been accessed recently. Branch Scope
: Caches are scoped by branch. A cache created on a feature branch is not accessible to other feature branches, but all branches can access the default branch Stack Overflow 3. Debugging Cache Misses
If your workflow reports a "Cache not found," check these three common failure points: Key Mismatch must be a perfect match. If you use a hash like hashFiles('package-lock.json')
, ensure that file actually exists at the time the cache step runs. Restore Keys : If a primary key isn't found, use restore-keys
to pull the most recent partial match. If this is missing, you will always start from scratch on a primary miss. Immutability : Once a cache is saved for a specific cannot be updated
. To "refresh" a cache, you must change the key name (e.g., by incrementing a version number in your YAML). GitHub Docs 4. Advanced CLI Debugging You can interact with the cache directly using the GitHub CLI ( gh-actions-cache extension. Stack Overflow gh actions-cache list View all caches in the terminal. gh actions-cache delete
The cache action creates a .tar archive. Debug logs reveal segmentation:
[debug] Compressing 1,234 files (245MB)
[debug] Archive segmented into 3 parts
[debug] Uploading part 1/3...
If you see a segmentation error (e.g., Part 2 failed to upload), you have a network issue or a file that changed during compression (race condition).
gh cache delete <cache-id>
actions/cache Debug OutputIf using actions/cache@v3, add the verbose input:
- name: Cache node_modules
uses: actions/cache@v3
with:
path: node_modules
key: $ runner.os -node-$ hashFiles('package-lock.json')
env:
CACHE_VERBOSE: true # Extra debug logs
| Symptom | Likely Cause | Debug Action |
|--------|--------------|---------------|
| Cache always misses | Dynamic key includes timestamp or commit SHA | Print key value in logs; remove non-deterministic parts |
| Restore succeeds but build fails | Cache contains platform-specific binaries (e.g., Linux binaries on macOS) | Check runner.os in key; separate caches per OS |
| Cache too large | Unnecessary files (logs, temp, downloads) | Exclude them via paths or use tar --exclude |
| Cache not deleted after PR merge | No automated cleanup | Use actions/cache with save-always: false or set a TTL via API |
| Cache restore slow | Many small files or deep nesting | Cache archives (e.g., .tar.zst) instead of directories |