Repository navigation
run-task hangs when cloning a github repository with LFS files and git-lfs installed #775
Description
We had an odd issue in some translations tasks last week: they suddenly started requiring credentials to clone the repository. It's unclear why this happened (the repository is still public...), but nevertheless it highlighted the fact that run-task simply hangs in this scenario, and wastes resources until the task times out:
[vcs 2025-09-09T18:43:02.107+00:00] executing ['git', 'clone', 'https://github.com/mozilla/translations', '/builds/worker/checkouts/vcs']
[vcs 2025-09-09T18:43:02.109+00:00] Cloning into '/builds/worker/checkouts/vcs'...
Username for 'https://github.com': [taskcluster 2025-09-09T18:59:21.223Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Mir70a0hRim0qvIfTjAcRw/0/on/us-west1-b/3099223815166478912/until/1757445561.163
[taskcluster 2025-09-09T19:16:21.206Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Mir70a0hRim0qvIfTjAcRw/0/on/us-west1-b/3099223815166478912/until/1757446581.188
...
...
...
[taskcluster 2025-09-15T10:08:41.854Z] Command ABORTED after 135h25m43.489856784s: process aborted
[taskcluster 2025-09-15T10:08:41.854Z] Average Available System Memory: 49.86 GiB
[taskcluster 2025-09-15T10:08:41.854Z] Average System Memory Used: 568.96 MiB
[taskcluster 2025-09-15T10:08:41.854Z] Peak System Memory Used: 886.15 MiB
[taskcluster 2025-09-15T10:08:41.854Z] Total System Memory: 50.98 GiB
[taskcluster 2025-09-15T10:08:41.854Z]
[taskcluster 2025-09-15T10:08:41.866Z] === Task Finished ===
We should find a way to detect this and abort.
Activity
[-]run-task hangs when cloning a repository that requires credentials[/-][+]run-task hangs when cloning a github repository with LFS files and git-lfs installed[/+]bhearsum commentedon Sep 15, 2025
As it turns out, this is related to git-lfs, although I haven't determined the root cause. https://github.com/mozilla/translations has had LFS files for ~10 months, and we only started having issues with them in the past ~5 days. mozilla/translations#1247 (comment) has a small amount of extra detail. It certainly seems like something has changed or regressed on the github side.