fetch() hangs with authenticated git repo and credential helper enabled-but-not-yet-started

[kartiksubbarao]

I'm running GitPython 0.3.2 RC1 with Python 2.7.6 on Ubuntu 14.04. My ~/.gitconfig has these lines:

[credential]
    helper = cache --timeout=3600

I've narrowed the problem down to this script:

#! /usr/bin/python

import git

repo = git.Repo.init("/tmp/repo1")
origin = repo.create_remote('origin', 'https://repohost.example.com/some/repo')
origin.fetch()

After I'm prompted for my username and password, the script hangs. If I remove my ~/.gitconfig, the script doesn't hang. Also, if the git-credential-cache--daemon process already happens to be running successfully, the script doesn't hang.

The broader context of this is that I'm running Salt with the GitPython backend and getting hangs if the credential helper daemon isn't already running.

[Byron]

Thanks for the elaborate description.

It seems that GitPython is not even opening stdin in the processes it instantiates, yet the sub-processes git-fetch opens seems to open stdin by default. Apparently they also don't care whether or not they are connected to a tty. I couldn't find any option or configuration that would prevent it from trying any prompts, which shouldn't mean such an option doesn't exist though.

From my point of view, there is nothing GitPython can do to help with such a situation. Please let me know if you think differently about it.

Thanks for your feedback.

[Byron]

You can watch the development stream on youtube.

GitPython #7 [issue 295 - research how to prevent password prompts]

[kartiksubbarao]

First -- thank you for taking the time to investigate this so thoughtfully. I haven't seen any developers screencast their work like this, it's an intriguing approach. I just wish the resolution was higher so that I could read the code clearly -- I could almost make out the characters :-)

I think you may be misunderstanding my bug report slightly. During the screencast, you were looking for ways to disable the password prompt. That's actually not what I want in this case. If I didn't want to be interactively prompted, I could pass "https://username:password@github.server" as the URL for the git repo, or use OAUTH or some other method of authentication.

In this case, I want to be prompted the first time for the username and password (and for the credentials helper to pick up that info so that I don't need to be subsequently re-prompted). The problem is that after my username and password are successfully read, the script hangs for some unknown reason.

I ran an strace on the process and could see that it transmitted my username and password to the git credentials cache daemon. But at that point the strace becomes difficult to follow. It looks like there is at least one git process that is in a defunct state, along with the credentials cache daemon. My guess is that there is some deadlock going on.

Does this help clarify what I'm seeing? I can also try running a more recent version of GitPython to see if that might fix the problem.

[kartiksubbarao]

I can confirm that the hang is still present in GitPython 1.0.1.

[kartiksubbarao]

Found something interesting while troubleshooting. I can reproduce the hang with the following code in a test script, which is akin to how GitPython invokes Popen():

proc = Popen('git fetch -v'.split(),
             stdin=None, stdout=PIPE, stderr=PIPE,
             close_fds=True)
(stdout_value, stderr_value) = proc.communicate()

If I then change stderr=PIPE to stderr=None, the hang goes away and the script runs fine. It would appear that python's subprocess.Popen() function may not be handling stderr in a way that plays well with the credentials helper.

I did some more searching and found a bug report in the vcstools project that looks identical to what I'm seeing here:

vcstools/vcstools#67

It looks like what they do is disable the stdout/stderr PIPE from the command to work around the issue. That might be too harsh. Another thought might be, as a configurable option, to selectively disable the stderr pipe for the git fetch command (and just rely on the exit status to tell success/failure). What do you think? I realize that GitPython is in no way responsible for the behavior of subprocess.Popen() -- just trying to see if there might be a way to enable this use case without introducing unwanted complications for GitPython.

[kartiksubbarao]

I found the root cause of the problem. The git credential-cache--daemon process wasn't closing stderr, so the Popen() function kept waiting to read from it. The good news is that this has been resolved as of git 2.2.0:

git/git@f5e3c0b

I was running the standard version of git (1.9.1) on Ubuntu 14.04. Installing a more recent version resolved this issue for me.

[Byron]

Great work ! You found the needle in the haystack by pinpointing the commit with the fix. I wonder how you were able to figure that out, and I would be very interested in learning about your thought-process. After all I wasn't able to arrive at this conclusion, as I couldn't see that far.

I just wish the resolution was higher so that I could read the code clearly -- I could almost make out the characters :-)

The issue you encountered is a specialty of youtube, which will make videos available in 360p or less initially, providing gradually higher resolution versions over time. All my videos are in 1080p@24fps, but it takes about an hour for the best resolution to actually show up.

[kartiksubbarao]

Glad to hear that the video issue is only temporary -- I was able to watch one of your earlier videos with much better clarity!

Here was the general flow of my thought process, with the caveat that there were various detours along the way as well:

I'm more fluent with Perl than I am with Python, so I wanted to see if I could reproduce the problem in Perl. The Perl code didn't hang. The most obvious difference was how stderr was being treated. In my first-pass Perl code, I was doing:

open(GITFETCH, "git fetch -v < /dev/null |");

This only opens the pipe to stdout and leaves stderr unchanged, whereas in the Python code stderr was being piped as well. This led me to my first workaround idea of setting stderr to None in the popen() call. It also gave me more precise keywords to search for, which is how I found the vcstools bug report -- I think I used something like "git credentials helper python popen stderr hang".

To zero in on this further, I fired up the python debugger on the script to see if I could trace into the Popen() call and see what was happening. Previously when I tried this, git-credential-cache--daemon started to compete with the python debugger when prompting for my password (since it opens /dev/tty directly). So I wasn't able to type anything into the debugging session after that point. I thought about ways to work around this limitation. After various searches and learning more about the git credentials helper, I discovered a way to reproduce the hang without having to type in keyboard input or even having to do a git fetch.

#! /bin/sh
echo "protocol=https\nhost=example.com\nusername=asdf\npassword=asdf123\n" | git credential approve

After saving the above into a script called gcf.sh, I could reproduce the hang by running the following python code:

proc = Popen('gcf.sh', stdin=None, stdout=PIPE, stderr=PIPE, close_fds=True)
(stdout_value, stderr_value) = proc.communicate()
print "gcf.sh: stdout=[%s] stderr=[%s]" % (stdout_value, stderr_value)

With the tty contention gone, I could now step through this code further with the python debugger. When I did, I found that poll() still wanted to read from stderr even after it was done with stdout. I ran lsof on the git-credential-cache--daemon process forked by gcf.sh and found that, sure enough, stderr was still connected to the pipe opened by the python code. I then went to the source code for this process, credential-cache--daemon.c, to see what it was doing with stdout/stderr:

https://github.com/git/git/blob/master/credential-cache--daemon.c

What immediately caught my eye was the summary commit message from the most recent commit, at the top of the page:

peff on Sep 16, 2014 credential-cache: close stderr in daemon process

When I read through the full commit log for this change and saw the change diff, I immediately knew that this was the fix. Of course I tested it to be sure :-)

The funny thing is, I had seen this very same summary commit message much earlier in my troubleshooting process, while browsing through the git code. But it didn't occur to me then that this was the solution.

[Byron]

Thank you so much for sharing your process ! I hope it helps others as much as me - I will be trying to think of it when I am willing to give up on finding the solution to an issue next time :).