Enable authenticated direct downloads in SCIO to fetch inputs from private sources

[pombredanne]

We should enable authenticated direct downloads in SCIO to fetch inputs from private sources

• See also Add integration with third party tools such as forges, repos, CI/CD, messaging and others tools #619

[pombredanne]

#1095 is a dupe:

How to access downloads that are password-protected?

I would like to create a pipeline and add input URLs for code that is either in a private repo or at a download that requires a password.
How can I do this?

I guess a workaround may be to:

1. Download separately my repo tarball using my credentials (with something like wget or curl) separately, or the same for any protected URL
2. Add the downloaded file to my project either through an API call or a command line call

[pombredanne]

For now, there is no support for fetching private sources in SCIO

For the implementation, let's discuss it there ... some directions and questions:

• .netrc as already used here https://github.com/nexB/python-inspector/blob/609a81ab6d0c36545f3e3516045b45523ad1e721/src/python_inspector/utils.py#L21
• environment variables
• JSON files (like created by docker/podman upon login)

Questions

• How to handle per-project authentication needs? Or is this always instance wide?
• Do we even want to store any credentials/secrets?
• Could we integrate with secret stores?
• For the specific use cases around GitHub and GitLab, could we reuse some of their OAuth and existing auth support?

[tdruez]

Implemented in #1097

See https://scancodeio.readthedocs.io/en/latest/faq.html#how-to-fetch-files-from-private-sources-and-protected-by-credentials and https://scancodeio.readthedocs.io/en/latest/application-settings.html#fetch-authentication