Skip to content

feat: implement hybrid cache architecture - #1246

Open
fabiovincenzi wants to merge 73 commits into
finos:mainfrom
fabiovincenzi:feature/hybrid-cache
Open

fabiovincenzi wants to merge 73 commits into
finos:mainfrom
fabiovincenzi:feature/hybrid-cache

Conversation

@fabiovincenzi

@fabiovincenzi fabiovincenzi commented Oct 13, 2025 •

Copy link
Copy Markdown
Contributor

Issue: #1207

Summary

Two-tier cache system for faster git operations: persistent bare repos + ephemeral working copies.

Architecture

  • .remote/cache/ - Shared bare repositories (persistent)
  • .remote/work/<push-id>/ - Per-push working copies (temporary)

Configuration

Add to proxy.config.json:

{
  "cache": {
    "maxSizeGB": 2,
    "maxRepositories": 50,
    "cacheDir": "./.remote/cache"
  }
}

Security

  • Each push gets unique working copy (.remote/work//)
  • Bare cache contains only git objects (no user files, no credentials)
  • sanitizeRepositoryConfig() immediately removes credentials from git config
  • Working copies deleted after push completes

@netlify

netlify Bot commented Oct 13, 2025 •

Copy link
Copy Markdown

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 61f8a02
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6afc2dacdfad0124a6201d1d
😎 Deploy Preview https://deploy-preview-1246.git-proxy.preview.finos.org
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Oct 13, 2025 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.28358% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.02%. Comparing base (9b2501d) to head (61f8a02).

Files with missing lines Patch % Lines
src/proxy/processors/push-action/cache-manager.ts 95.00% 4 Missing and 1 partial ⚠️
...rc/proxy/processors/push-action/PullRemoteHTTPS.ts 0.00% 4 Missing ⚠️
src/proxy/processors/push-action/git-operations.ts 94.73% 0 Missing and 3 partials ⚠️
src/proxy/processors/push-action/PullRemoteSSH.ts 89.47% 2 Missing ⚠️
src/proxy/processors/push-action/metrics.ts 84.61% 0 Missing and 2 partials ⚠️
...roxy/processors/push-action/PullRemoteSSHCached.ts 91.66% 0 Missing and 1 partial ⚠️
src/proxy/processors/push-action/cachedClone.ts 98.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1246      +/-   ##
==========================================
+ Coverage   88.87%   89.02%   +0.14%     
==========================================
  Files         127      133       +6     
  Lines        7092     7350     +258     
  Branches     1318     1366      +48     
==========================================
+ Hits         6303     6543     +240     
- Misses        493      504      +11     
- Partials      296      303       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@fabiovincenzi
fabiovincenzi marked this pull request as ready for review October 22, 2025 14:13

@06kellyjac 06kellyjac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm looking forward to faster pulls!

Caches are a common place for logic or security problems so we may want added scrutiny and testing for this PR to be extra safe :)

Added some thoughts, not ran this locally just yet.

Comment on lines +10 to +22
if (process.env.NODE_ENV === 'test') {
// TEST: Full cleanup (bare cache + all working copies)
try {
if (fs.existsSync('./.remote')) {
fs.rmSync('./.remote', { recursive: true, force: true });
step.log('Test environment: Full .remote directory cleaned');
} else {
step.log('Test environment: .remote directory already clean');
}
} catch (err) {
step.log(`Warning: Could not clean .remote directory: ${err}`);
}
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like a safer option that ensures what we're testing more closely matches prod would be to either do some separate cleanup as necessary in the tests, or adjust config to have a much lower max (or even 0) then we can see if CacheManager itself is doing the cleanup.

Comment thread src/config/index.ts Resolved

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a code review for now - I'll be testing the code later!

I was wondering if you could do some profiling to verify/document the speed improvements with this PR. Would be nice to try it out with massive repos (for example https://github.com/backstage/backstage which is >10GB) to see the difference.

I'll try to do some profiling soon as well 🙂

@06kellyjac

Copy link
Copy Markdown
Contributor

why on earth is backstage's repo 10GB? 😭
Even nixpkgs is only 6.1GB ATM on my machine

@jescalada

Copy link
Copy Markdown
Contributor

@06kellyjac It might be because of the tons of plugins that come with the repo (many of which have redundant components). Pulling with git clone --filter=blob:none https://github.com/backstage/backstage.git seems to make things a lot more speedy, however I'm not sure if we want to do things that way in the pullRemote. Perhaps a config option to filter out the BLOBs could help with these massive repos?

@fabiovincenzi

Copy link
Copy Markdown
Contributor Author

Merged latest main. Since this PR was opened, pullRemote has been split into PullRemoteBase / PullRemoteHTTPS / PullRemoteSSH, so the cache now plugs into those classes instead of replacing pullRemote.

Three things for review:

  • the cache is now optional via cache.enabled, with it off, the factory returns the existing implementations untouched
  • it works over SSH as well as HTTPS, sharing the same mirror
  • the cache key now includes the project, previously collided on a single mirror and fetched each other's history into it

Verified end-to-end on both protocols (cache MISS then HIT).

@kriswest kriswest modified the milestones: 2.3.0, 2.4.0 Oct 5, 2026
@kriswest kriswest removed this from the 2.4.0 milestone Nov 2, 2026

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more comments from me (and Claude). 👍🏼

Also, I think it'd be great to document this a bit better (for example in website/docs/deployment), so users (and agents) are aware of the caching feature even if it's disabled by default,

env?: NodeJS.ProcessEnv,
): Promise<{ status: number; stderr: string }> {
return new Promise((resolve, reject) => {
const proc = spawn('git', args, { stdio: ['ignore', 'ignore', 'pipe'], env: buildEnv(env) });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a git fetch, for example, were to hang? Doesn't this still block subsequent pushes to the same repo because of withRepositoryLock?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, fixed! git commands now time out after cache.gitTimeoutSeconds

*/
export function cacheKeyFor(action: Action): string {
const sanitise = (value: string) => value.replace(/[^\w.-]+/g, '-');
return [hostOf(action.url), action.project, action.repoName]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review finding:

The key can still collide. The sanitiser turns / into - and keeps _, so GitLab a/b/c and a-b/c both map to gitlab.com__a-b__c.git, and x__y/z and x/y__z both map to github.com__x__y__z.git

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the key now ends with a short hash of the unsanitised host/project/repo

Comment thread config.schema.json Outdated
"description": "Directory path for bare repository cache (default ./.remote/cache)"
}
},
"required": ["maxSizeGB", "maxRepositories", "cacheDir"],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if these have to be required - we already have defaults for them in cache-manager.ts, unless the point is to make these values more visible and obviously "editable". Just a nitpick though, feel free to ignore 👍🏼

// Dot-prefixed folders are staging clones still being built, not cached repos
if (entry.isDirectory() && !entry.name.startsWith('.')) {
const repoPath = path.join(this.repoCacheDir, entry.name);
const sizeBytes = this.getDirectorySize(repoPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review finding:

getDirectorySize walks every mirror synchronously after each push, inside the global mutex. That blocks the event loop, which contradicts the README’s “never blocks the event loop” line. A cached size per repo, refreshed only for the repo that just changed, would be enough.

if (existed) {
try {
await gitOps.fetch({
dir: bareRepo,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another AI finding:

Any fetch failure, including a transient network error, triggers a full bare re-clone while the lock is held. Rebuilding only on errors that look like corruption, and failing the push otherwise, would avoid surprise multi-gigabyte clones.

Comment thread config.schema.json Outdated
"properties": {
"enabled": {
"type": "boolean",
"description": "Enable the bare repository cache. When disabled, each push clones directly from the remote as usual.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to stress the importance of the caching feature while keeping it disabled by default

Suggested change
"description": "Enable the bare repository cache. When disabled, each push clones directly from the remote as usual.",
"description": "Enable the bare repository cache (recommended). When disabled, each push clones directly from the remote as usual.",

Comment thread proxy.config.json
]
},
"cache": {
"enabled": false,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with keeping this disabled by default, but we do want to advertise the caching capabilities to new users (and AI agents figuring out how to deploy/set up GitProxy).

Perhaps we could highlight this throughout the docs, deployment guide, etc.?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve push flexibility and pull times

6 participants