Skip to content

ci(cache): delete BuildKit cache blobs older than 30 days, weekly - #477

Merged
sunib merged 1 commit into
mainfrom
ci/buildkit-cache-cleanup
Nov 9, 2026
Merged

sunib merged 1 commit into
mainfrom
ci/buildkit-cache-cleanup

Conversation

@sunib

@sunib sunib commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

Since at least Nov 5, every image build failed: the CI base container, the instrumented project image and both release images, on main and on PRs (#474). They all failed on one GitHub Actions cache entry, 1257206725, with BlobNotFound. That entry held BuildKit's empty layer (buildkit-blob-1-sha256:4f4fb700…) and was created on 2025-09-30. GitHub evicts an entry only after a week without use, and every build uses the empty layer, so the entry was never evicted. When GitHub lost the file behind it, BuildKit's type=gha cache read failed the build instead of rebuilding the layer. Deleting the entry by hand fixed it.

What this does

A new hygiene workflow, .github/workflows/cache-cleanup.yml, runs weekly (Monday 02:23 UTC) and on demand. It deletes buildkit-blob-* cache entries created more than 30 days ago, so no entry lives long enough to cause this again for long. Started by hand, it takes max_age_days and a dry_run switch.

  • It only touches buildkit-blob-* keys. BuildKit's index entries and the Trivy database cache are left alone.
  • Cost: once a month, the next image build on main rebuilds the deleted layers and writes them back. PR builds only read the cache, so a PR opened in that window builds those layers itself.
  • The job needs actions: write to delete cache entries and nothing else. It needs no checkout.
  • It fails if any delete fails, so a lost permission doesn't go unnoticed.

docs/ci-overview.md lists it with the other hygiene workflows.

Validation

  • task lint (includes actionlint) and task test pass.
  • Ran the list-and-filter step locally against the live cache: 0 of 212 BuildKit blobs are older than 30 days today, now that the broken entry is gone.
  • After merge: start it once by hand with dry_run: true to confirm the token can list caches.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Added a scheduled weekly cleanup for older image-build cache entries, with options to run it manually, adjust the age threshold, and preview changes without deleting entries. Results are reported, and failed deletions cause the run to fail.
  • Documentation
    • Updated the CI overview to describe the cache cleanup workflow.

GitHub evicts a cache entry only after a week without use, so a layer every image build reuses kept one entry for over a year. When GitHub lost the file behind it, every image build on main and on PRs failed with BlobNotFound until the entry was deleted by hand. A weekly workflow now deletes buildkit-blob-* entries created more than 30 days ago (adjustable, with a dry run, when started by hand), which bounds how long a lost blob can break the builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The pull request adds a scheduled and manually triggered workflow to clean up old BuildKit cache entries. It supports an age threshold and dry-run mode, reports results, and documents the workflow in the CI overview.

Changes: BuildKit cache cleanup

Layer / File(s) Summary
Workflow triggers and access
.github/workflows/cache-cleanup.yml
The workflow runs on Mondays at 02:23 UTC or by manual dispatch. Manual runs accept a max_age_days value and a dry_run setting. The workflow defines repository and Actions permissions and serializes runs.
Cache cleanup and reporting
.github/workflows/cache-cleanup.yml, docs/ci-overview.md
The job lists up to 5,000 buildkit-blob-* entries older than the UTC cutoff. It skips deletion in dry-run mode, reports results, and fails if a deletion fails. The CI overview describes the workflow and lists it under hygiene workflows.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant GitHubTrigger
  participant CacheCleanupWorkflow
  participant ActionsCache
  GitHubTrigger->>CacheCleanupWorkflow: Start scheduled or manual run
  CacheCleanupWorkflow->>ActionsCache: List matching old BuildKit cache entries
  ActionsCache-->>CacheCleanupWorkflow: Return matching entries
  CacheCleanupWorkflow->>ActionsCache: Delete entries unless dry-run is enabled
Loading
Merge Risk: Minimal · up to b6c28

This change adds a weekly cleanup that removes old image-build cache blobs. When a deleted blob is still referenced by a cache index, the build treats it as a cache miss and rebuilds that step instead of failing. No merge-blocking risk remains.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly summarizes the main change: a weekly cleanup of BuildKit cache blobs older than 30 days.
Description check Passed The description explains the problem, the workflow behavior, its permissions and scope, documentation changes, and validation. It does not use the template headings or complete its checklists, but it …
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the problem, the workflow behavior, its permissions and scope, documentation changes, and validation. It does not use the template headings or complete its checklists, but it provides the main information those sections request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9d19b051-2085-414b-b300-d8f0b315130f
Commits

Reviewing files that changed from the base of the PR and between f48cba0 and b6c282c.

Files selected for processing: 2

  • .github/workflows/cache-cleanup.yml
  • docs/ci-overview.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@sunib
sunib merged commit 397a4e0 into main Nov 9, 2026
23 checks passed
@sunib
sunib deleted the ci/buildkit-cache-cleanup branch November 9, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant