Dev utility: build and download the sdist and wheels of a single package and all its dependencies

[pombredanne]

As a developer of ScanCode, I want to download and or build as needed the wheels and sources of a given package or package/version. For this I expect to call a command line tool that would accept a package name or name==version as an argument and a target directory where the built wheels and tarballs would be downloaded.

As an output of this command, I expect to find in the target directory:

1. pre-built wheels
   1.1. either one for each OS/Python version combo when the wheel contains native
   1.2. or one for each Python version if the wheel is Python-specific but not OS-specific
   1.3 or one for each OS if the wheel is OS-specific but not Python-specific

2. the corresponding source tarball or zip for each wheel (unless the wheel is a pure Python wheel that contains all its source code in which case this may not be needed)

3. one .ABOUT file for each wheel or source tarball/zip with all found details and URL
   NB: Since there is no place to download that from , it may need to be created from scratch as a placeholder empty file? TBD

4. corresponding NOTICE and LICENSE files
   NB: Since there is no place to download that from, it may need to be created from scratch as a placeholder empty file? TBD

[pombredanne]

@Abhishek-Dev09 think about:

1. pip download and its flags to select a proper source and binary version such as --platform, --python-version, --implementation --abi --no-binary --only-binary
2. what you are doing with the pombreda/thirdparty* repos ... for the build that would need to be eventually automated and/or documented... or you could use romp? TBD

[pombredanne]

See #2118

[pombredanne]

This is going to be using the utilities listed in #2017 such a romp or https://github.com/pombreda/thirdparty-manylinux/ and https://github.com/pombreda/thirdparty

1. we need to provide a list of packages that need to be built as an input and IDEALLY that should be the same as our requirements file
2. the utilities either need to upload directly to our upcoming PyPI "GitHub release" repo at Dev utility: Create new repo to expose pre-built wheels, tarballs and ABOUT files as releases in a PyPI-like repo #2068 OR would fetch these locally and will be uploaded after that to the PyPI "GitHub release" using the github_release script in Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117
3. other files added such ABOUT files and license would be uploaded to the PyPI "GitHub release" using the github_release script in Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

[pombredanne]

Eventually we built a tool that handles creation, documentation and publishing of wheels and source https://github.com/nexB/scancode-toolkit/tree/1bddb92c70abcf26cf9435061e429bae62cbe040/etc/release
And this is also part of the https://github.com/nexB/skeleton

Using romp it builds wheel and tests them on many OS/Python combos

In the end they end up pushed to https://thirdparty.aboutcode.org/pypi/