De-vendor prebuilt binaries to ease packaging for Linux distros

[pombredanne]

@pabs3 and @matthieucan suggested to me on the OFTC #debian-qa IRC channel that ScanCode could be packaged as a Debian Linux package.

@dvc94ch in #288 looks for help on packaging ScanCode for the Guix Linux distro.

Packaging in a distro should be a non-event but it would be a tad involved today.
ScanCode is already buildable as a wheel which is a first good step towards this.

As a second step, any pre-built binaries that are vendored as a user convenience in the src/*/bin directories should be de-vendored from main source tree and provided instead either:

• as another Python package (which can be vendored again such that the current user experience to be able to just download or clone and run without any other install required still works fine)
• as distro-provided dependent packages such that a distro packaging of ScanCode can use proper deps on other distro packages in this case

This way a distro packaging would be vastly simplified

[pabs3]

Generally, I suggest a good way to replace embedded code copies is to create binary packages for Windows/macOS/etc and only embed deps in the binary packages that can't be installed via binary package deps. I think Python has some sort of system for binary packages too. To satisfy the the folks who want to install from source with one command you can make the setup.py install all dependencies, either via their distro or via the relevant language package managers.

[pombredanne]

@pabs3 wrote

Generally, I suggest a good way to replace embedded code copies is to create binary packages for Windows/macOS/etc and only embed deps in the binary packages that can't be installed via binary package deps.

I agree this is the way to go.

Python has a system for binary packages alright and I should be able to use that to include plain binaries for the few pure native exe and libs (such as libarchive, libmagic, p7zip and a few more) as extra packages.

And I can make these such that we have variants for Windows, Mac and Linux32 and 64 instead of the vendored directories at https://github.com/nexB/scancode-toolkit/tree/develop/src/typecode/bin or https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode/bin ...

I still want to keep the experience to download and run without having anything other dep needed beside Python proper and that on all three OSes and this should work alright: I wish that dealing with installing dependent packages from sources or prebuilt binaries on Windows and Mac would be a well solved problem, but this is not the case.

So the way could be to break these binaries apart such that:

1. they would still be available but provided in some optional Python package such that the current experience of fetch or clone and run without build-essential like deps can be kept the same

2. when packaged for a distro or when building from sources, you could use instead the corresponding distro packages (such as https://packages.debian.org/jessie/libarchive13 or https://packages.debian.org/jessie/p7zip or https://packages.debian.org/jessie/file and a few more)

One difficulty is that the available Debian versions of these deps may not match the exact version that ScanCode expects... but that should not be a major issue to deal with (though it will need testing for sure)

[pombredanne]

@roscopecoltran for your issue #636 related to the pre-built binaries it is best to do this here

[pombredanne]

This is also needed for the work of @maxyz
The approach would likely to extend scancode's new plugin mechanisms by @yashdsaraf to provide these binaries as optional plugins. (e.g. optional Python packages) or get them from the PATH as a last resort.

[bhavishyagopesh]

@pombredanne So is this done now, (I mean a PyPI package containing all binaries)...I would like to work on this packaging problem,( and eventually #487 ).

[pombredanne]

@bhavishyagopesh Thanks! No this is not done.
The gist of it: check how the commoncode.command code works and how there are bundled pre-built binaries in bin/ dirs in extractcode and typecode.

Ideally, we want a plugin system using the same approach as the scancode plugins with pluggy such that a plugin can provide a certain binary for an os/arch and that this is usable by the code instead of the bundled binaries. Then if no plugin provides a binary, a command or DLL should be used from the PATH as a last resort. This way a certain binary can either be provided with ScanCode as a plugin or not be provided at all and obtained from the PATH as provided by a distro for intance.

[pombredanne]

@bhavishyagopesh any progress on your side? I just got a new report on a related issue in #834 by @spbkelt so since you want to work on this issue, I'd need to know what's up and what could be an ETA?

[bhavishyagopesh]

@pombredanne I went through ur suggestion, I need to confirm a few things :

1. Where do we intend to keep the binaries eventually.
2. So almost all platform(os/arch) detection is being done by commoncode.system and we just need to fetch the matching binaries using plugin similar to commoncode.command.
3. Where will the plugin code go, in a separate repo?...

Thanks for your patience.