git lfs support

Does libgit2sharp support working with repo that contains github lfs binding. If so, where i can find any info how to use it?

Git LFS is implemented via smudge/clean filters - it's not supported out-of-the-box but is possible to do.

Thanks for answer.

It seems that GitHub client works that way. It would be nice to have the support of the git lfs out of the box because the approach with smudge/clean filters looks very complex.

It seems that GitHub client works that way. It would be nice to have the support of the git lfs out of the box because the approach with smudge/clean filters looks very complex.

I pushed a change today which should make it easier, but not "out of the box".

Any progress on this? Would love to have this supported.

Any progress on this? Would love to have this supported.

It is rather trivial to support if Git+Bash are on the box. Remember that Git's filters and hooks are both assumed to be Borne Again Shell Scripts (BASH), so there's no real way around having Bash on the box. Assuming it's a Linux box, you can be assured it is there.

What can I do to get this working?
Anyone know of a workaround or what code needs to be changed to make this work?
I started making a GUI tool thinking this would just work....

What can I do to get this working?

Just register your callback handler with GlobalSettings (https://github.com/libgit2/libgit2sharp/blob/master/LibGit2Sharp/GlobalSettings.cs#L208).

Then use that callback to send the appropriate commands to Bash.

Thanks but what git-lfs methods do I invoke on Clean, Complete, Create, etc?
They're being invoked but I'm not sure how this correlates to "git-lfs add" etc?
The docs in this area on the libgit2 site don't have any use cases...

So far I'm doing stuff like this without any luck:

using (var process = new Process())
            {
                process.StartInfo.FileName = "cmd";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.CreateNoWindow = false;
                process.StartInfo.UseShellExecute = false;
                process.Start();

                process.StandardInput.WriteLine("git-lfs clean \"" + path + "\"");
                process.StandardInput.Flush();

                process.StandardInput.Close();
                process.WaitForExit();
            }

What commands are "appropriate" and do I still need to call the base virtual method "base.Clean(...)"?

So I have it working on small files where "Clean" only needs to be called once.
If it needs to be called multiple times "git-lfs status" doesn't show the file as staged.

Here is the code (any ideas?)

protected override void Clean(string path, string root, Stream input, Stream output)
        {
            //base.Clean(path, root, input, output);

            using (var process = new Process())
            {
                process.StartInfo.FileName = "git-lfs";
                process.StartInfo.Arguments = "clean";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.RedirectStandardOutput = true;
                process.StartInfo.CreateNoWindow = true;
                process.StartInfo.UseShellExecute = false;
                process.Start();

                input.CopyTo(process.StandardInput.BaseStream);
                input.Flush();
                process.StandardInput.Flush();
                process.StandardInput.Close();
                input.Close();

                process.WaitForExit();

                process.StandardOutput.BaseStream.CopyTo(output);
                process.StandardOutput.BaseStream.Flush();
                output.Flush();
                output.Close();
                process.StandardOutput.Close();
            }
        }

        protected override void Complete(string path, string root, Stream output)
        {
            output.Close();
            //base.Complete(path, root, output);
        }

More likely what you should be doing is passing the clean of filter text to Git\bin\sh.exe and sending the stream to its stdin while collecting the results from stdout.

Ok have it commiting to lfs correctly via the code below.

However when I call "var blob = RepoUserControl.repo.ObjectDatabase.CreateBlob(item.FilePath);" its invoking "clean" which corrupts git-lfs as this is just used for viewing a file not commiting.

public class LFSFilter : Filter
    {
        private Process process;

        public LFSFilter(string name, IEnumerable<FilterAttributeEntry> attributes) : base(name, attributes)
        {
        }

        protected override void Clean(string path, string root, Stream input, Stream output)
        {
            try
            {
                // write all file data to stdin
                input.CopyTo(process.StandardInput.BaseStream);
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Complete(string path, string root, Stream output)
        {
            try
            {
                // finalize stdin and wait for git-lfs to finish
                process.StandardInput.Flush();
                process.StandardInput.Close();
                process.WaitForExit();

                // write git-lfs pointer for 'clean' to git or file data for 'smudge' to working copy
                process.StandardOutput.BaseStream.CopyTo(output);
                output.Flush();
                output.Close();

                process.Dispose();
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Create(string path, string root, FilterMode mode)
        {
            try
            {
                // launch git-lfs
                process = new Process();
                process.StartInfo.FileName = "git-lfs";
                process.StartInfo.Arguments = mode == FilterMode.Clean ? "clean" : "smudge";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.RedirectStandardOutput = true;
                process.StartInfo.CreateNoWindow = true;
                process.StartInfo.UseShellExecute = false;
                process.Start();
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Initialize()
        {
            base.Initialize();
        }

        protected override void Smudge(string path, string root, Stream input, Stream output)
        {
            try
            {
                // write git-lfs pointer to stdin
                input.CopyTo(process.StandardInput.BaseStream);
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }
    }

Here is my filter so far. It works in most cases (checkouts, merges, etc) BUT for some reason when cloning a repo some objects get there file data replaced with lfs pointers: https://github.com/zezba9000/Git-Game-GUI/blob/master/GitGameGUI/RepoUserControl.xaml.cs#L22

Why or why didn't git just expose Git-Core as a lib people could interface with... major design flaw with git.

Was the filter invoked for those objects? If so, was git-lfs called for those objects? If so, did it succeed or fail? What was its output?

Yes its called for all of my objects under lfs. I get no exceptions or errors that I can see.
From what I can tell so far is maybe "Smudge" is being called as a "Clean" as the Output string is the working file and so I'm writing the pointer information to what should be file data.

So in short maybe "mode = Clean" when it should be "mode = Smudge". Maybe this is totally off, I'm very confused as to why this is happening.

One more important note! when I call "git-lfs smudge" and finish with "process.StandardInput.Close();" it doesn't trigger git-lfs to quit like it does with when I call "git-lfs clean" is called.

This is being testing with GitLabs running on Ubuntu 16.04.

@technoweenie

No - I'm afraid that I do not know what that does.

@technoweenie is this a command that is needed for normal usage?

Cool so pushing now works via this method: https://github.com/zezba9000/Git-Game-GUI/blob/master/GitGameGUI/ChangesUserControl.xaml.cs#L409

Just hope i'm not missing a condition with "git lfs pointer".

Nah, it's more of a utility command: https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-pointer.1.ronn

Awesome tnx!

@technoweenie

Great! Thanks @technoweenie, and @whoisj for the info! Closing this.

@zezba9000

One more important note! when I call "git-lfs smudge" and finish with "process.StandardInput.Close();" it doesn't trigger git-lfs to quit like it does with when I call "git-lfs clean" is called.

I got smudge to exit nicely just as clean do.

It seems that you have to consume the StandardOutput before waiting to exit.

Like so

protected override void Complete(string path, string root, Stream output)
    ...
    process.StandardInput.Flush();
    process.StandardInput.Close();
    process.StandardOutput.BaseStream.CopyTo(output);
    process.WaitForExit();

Thank you @zezba9000 - this made supporting lfs in my tool real easy.

Just remind someone else who is using this way to support lfs.

When you use git-lfs pre-push origin to push the objects to pushed, please check the source/destination of the commit sha.

In my case, the Destination is my local sha, and the Source is the remote one. That takes my whole day. :(

And you need to write the local at the first, with remote followed. See https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-pre-push.adoc

LibGit2Sharp Version: 0.26.2

@zezba9000

Thanks for the snippet @zezba9000 It does work nicely, but it is very slow due to having to launch a new Process for every file.
It is much more efficient to use the git lfs filter-process.
I posted an example here https://gist.github.com/dedmen/ab740ad9ebfde0403e8223480bef91ae

It only needs to launch one process (per repository) and processes all files with it. That runs much faster, especially after doing a pull where it goes over all the files with a quick clean.

@dedmen

@dedmen This is much newer code base FYI: https://github.com/reignstudios/Git-It-GUI
Now with Github Desktop though there has been less of reason to continue the project.

@zezba9000

Thanks for the snippet @zezba9000 It does work nicely, but it is very slow due to having to launch a new Process for every file. It is much more efficient to use the git lfs filter-process. I posted an example here https://gist.github.com/dedmen/ab740ad9ebfde0403e8223480bef91ae

It only needs to launch one process (per repository) and processes all files with it. That runs much faster, especially after doing a pull where it goes over all the files with a quick clean.

Hi @dedmen, thank you so much, this was exactly what I was looking for, I managed to integrate (maybe as of now, still a bit hackish) your Gist in git-tfs, which unfortunately lacks git-LFS (or I think, any kind of filtering) awareness. I tried it out after setting up a TFS repository containing also binaries which in the git repo cloned from it, are tracked by LFS (via the .gitattributes) and your LFSFilter is nicely running "Clean" on tracked binaries, before libgit2sahrp which is used by git-tfs creates a git commit.

Unfortunately, it's another story with binaries commited in git which git-tfs should then send to TFS. Running the command which will do that would checkout the commit to be transferred to TFS into a temporary "TFS workspace", which is just a folder containing the changes from the commit, but not a diff, but as their actual content, to let TFS know about it and do its own "diff magic" if applicable. Here I would have expected the LFSFilter to "smudge", but it actually "cleaned", therefore git LFS pointer files have been checked in to TFS, instead of the actual blobs.
Might there be still something incomplete with the implemetation from the gist, or do I have to further investigate what git-tfs foes in this scenario?
Maybe I would need to add more console traces and turn on some environment variables for debugging libgit2(sharp) to understand what'S going on...

I'm running my code in a git-SVN sync tool.
I'm pretty sure my checkout works fine, but I also do a SVN checkout right after which would overwrite the files if they were bad.
So I can't confidently say that my checkout is working.
Are you calling the PostCheckout() method from my snippet? It's probably not easy to see how to call it. I'll come back to this on Monday and post a sample on how to use that.

The filter code, just does what it gets told to do by libgit2. So there shouldn't be an issue in the filter itself. But maybe the post checkout is needed.

I guess you're right, I traced all method calls of your filter and only Initialize, Create, Clean and Complete are being called in general, when I fetch / pull data from TFS, also it's the only occasion I've seen Smudge being called, I would have expected to see it when checking in to TFS (so in the other direction). None of PrePush/PostCheckout/PostCommit have been called, I too suspected that could be the issue, you seem to confirm by emphasizing PostCheckout needs to be called...

@dedmen

I figured it out how to make it work when checking in to TFS as well, it was just changing one line in the git-tfs code. I will fork git-tfs and publish my changes, which I would eventually PR upstream, if it's ok with you to also incorporate your Gist, @dedmen ?

I didn't realize I forgot to put a license onto it.
Yeah do whatever with it!

Navigation Menu

Repository navigation

git lfs support #1236

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

git lfs support #1236

Description

Activity

shiftkey commented on Nov 30, 2015

Xarlot commented on Dec 1, 2015

whoisj commented on Dec 1, 2015

ejsmith commented on Feb 7, 2016

whoisj commented on May 9, 2016

zezba9000 commented on Jun 19, 2016

whoisj commented on Jun 20, 2016

zezba9000 commented on Jun 22, 2016

zezba9000 commented on Jun 22, 2016

zezba9000 commented on Jun 22, 2016

whoisj commented on Jun 22, 2016

zezba9000 commented on Jun 23, 2016

zezba9000 commented on Jun 23, 2016

ethomson commented on Jun 23, 2016

zezba9000 commented on Jun 23, 2016

6 remaining items

ethomson commented on Jun 24, 2016

zezba9000 commented on Jun 25, 2016

technoweenie commented on Jun 25, 2016

zezba9000 commented on Jun 25, 2016

ethomson commented on Jun 25, 2016

tedlofgren commented on Jun 30, 2020

orrindeng commented on Mar 30, 2023

dedmen commented on Jun 7, 2024

zezba9000 commented on Jun 7, 2024

lucianm commented on Jul 19, 2024

dedmen commented on Jul 19, 2024

lucianm commented on Jul 19, 2024

lucianm commented on Jul 20, 2024

dedmen commented on Jul 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions