Repository navigation
feat: implement hybrid cache architecture - #1246
fabiovincenzi wants to merge 73 commits into
Conversation
✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1246 +/- ##
==========================================
+ Coverage 88.87% 89.02% +0.14%
==========================================
Files 127 133 +6
Lines 7092 7350 +258
Branches 1318 1366 +48
==========================================
+ Hits 6303 6543 +240
- Misses 493 504 +11
- Partials 296 303 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
12b56bd to
734621c
Compare
Signed-off-by: Fabio Vincenzi <93596376+fabiovincenzi@users.noreply.github.com>
…zi/git-proxy into feature/hybrid-cache
06kellyjac
left a comment
There was a problem hiding this comment.
I'm looking forward to faster pulls!
Caches are a common place for logic or security problems so we may want added scrutiny and testing for this PR to be extra safe :)
Added some thoughts, not ran this locally just yet.
| if (process.env.NODE_ENV === 'test') { | ||
| // TEST: Full cleanup (bare cache + all working copies) | ||
| try { | ||
| if (fs.existsSync('./.remote')) { | ||
| fs.rmSync('./.remote', { recursive: true, force: true }); | ||
| step.log('Test environment: Full .remote directory cleaned'); | ||
| } else { | ||
| step.log('Test environment: .remote directory already clean'); | ||
| } | ||
| } catch (err) { | ||
| step.log(`Warning: Could not clean .remote directory: ${err}`); | ||
| } | ||
| } else { |
There was a problem hiding this comment.
I feel like a safer option that ensures what we're testing more closely matches prod would be to either do some separate cleanup as necessary in the tests, or adjust config to have a much lower max (or even 0) then we can see if CacheManager itself is doing the cleanup.
jescalada
left a comment
There was a problem hiding this comment.
Just a code review for now - I'll be testing the code later!
I was wondering if you could do some profiling to verify/document the speed improvements with this PR. Would be nice to try it out with massive repos (for example https://github.com/backstage/backstage which is >10GB) to see the difference.
I'll try to do some profiling soon as well 🙂
|
why on earth is backstage's repo 10GB? 😭 |
|
@06kellyjac It might be because of the tons of plugins that come with the repo (many of which have redundant components). Pulling with |
|
Merged latest main. Since this PR was opened, pullRemote has been split into PullRemoteBase / PullRemoteHTTPS / PullRemoteSSH, so the cache now plugs into those classes instead of replacing pullRemote. Three things for review:
Verified end-to-end on both protocols (cache MISS then HIT). |
jescalada
left a comment
There was a problem hiding this comment.
Just a few more comments from me (and Claude). 👍🏼
Also, I think it'd be great to document this a bit better (for example in website/docs/deployment), so users (and agents) are aware of the caching feature even if it's disabled by default,
| env?: NodeJS.ProcessEnv, | ||
| ): Promise<{ status: number; stderr: string }> { | ||
| return new Promise((resolve, reject) => { | ||
| const proc = spawn('git', args, { stdio: ['ignore', 'ignore', 'pipe'], env: buildEnv(env) }); |
There was a problem hiding this comment.
What happens if a git fetch, for example, were to hang? Doesn't this still block subsequent pushes to the same repo because of withRepositoryLock?
There was a problem hiding this comment.
Good point, fixed! git commands now time out after cache.gitTimeoutSeconds
| */ | ||
| export function cacheKeyFor(action: Action): string { | ||
| const sanitise = (value: string) => value.replace(/[^\w.-]+/g, '-'); | ||
| return [hostOf(action.url), action.project, action.repoName] |
There was a problem hiding this comment.
AI review finding:
The key can still collide. The sanitiser turns
/into-and keeps_, so GitLaba/b/canda-b/cboth map togitlab.com__a-b__c.git, andx__y/zandx/y__zboth map togithub.com__x__y__z.git
There was a problem hiding this comment.
the key now ends with a short hash of the unsanitised host/project/repo
| "description": "Directory path for bare repository cache (default ./.remote/cache)" | ||
| } | ||
| }, | ||
| "required": ["maxSizeGB", "maxRepositories", "cacheDir"], |
There was a problem hiding this comment.
Wondering if these have to be required - we already have defaults for them in cache-manager.ts, unless the point is to make these values more visible and obviously "editable". Just a nitpick though, feel free to ignore 👍🏼
| // Dot-prefixed folders are staging clones still being built, not cached repos | ||
| if (entry.isDirectory() && !entry.name.startsWith('.')) { | ||
| const repoPath = path.join(this.repoCacheDir, entry.name); | ||
| const sizeBytes = this.getDirectorySize(repoPath); |
There was a problem hiding this comment.
AI review finding:
getDirectorySizewalks every mirror synchronously after each push, inside the global mutex. That blocks the event loop, which contradicts the README’s “never blocks the event loop” line. A cached size per repo, refreshed only for the repo that just changed, would be enough.
| if (existed) { | ||
| try { | ||
| await gitOps.fetch({ | ||
| dir: bareRepo, |
There was a problem hiding this comment.
Another AI finding:
Any fetch failure, including a transient network error, triggers a full bare re-clone while the lock is held. Rebuilding only on errors that look like corruption, and failing the push otherwise, would avoid surprise multi-gigabyte clones.
| "properties": { | ||
| "enabled": { | ||
| "type": "boolean", | ||
| "description": "Enable the bare repository cache. When disabled, each push clones directly from the remote as usual.", |
There was a problem hiding this comment.
Just to stress the importance of the caching feature while keeping it disabled by default
| "description": "Enable the bare repository cache. When disabled, each push clones directly from the remote as usual.", | |
| "description": "Enable the bare repository cache (recommended). When disabled, each push clones directly from the remote as usual.", |
| ] | ||
| }, | ||
| "cache": { | ||
| "enabled": false, |
There was a problem hiding this comment.
I agree with keeping this disabled by default, but we do want to advertise the caching capabilities to new users (and AI agents figuring out how to deploy/set up GitProxy).
Perhaps we could highlight this throughout the docs, deployment guide, etc.?
Issue: #1207
Summary
Two-tier cache system for faster git operations: persistent bare repos + ephemeral working copies.
Architecture
.remote/cache/- Shared bare repositories (persistent).remote/work/<push-id>/- Per-push working copies (temporary)Configuration
Add to
proxy.config.json:{ "cache": { "maxSizeGB": 2, "maxRepositories": 50, "cacheDir": "./.remote/cache" } }Security