does scancode provide details from .deb or .rpm installed files?

[heckj]

Good afternoon,

I was attempting to utilize scancode-toolkit to see if I could get a list of all the relevant licenses as they apply to OS level packages inside a VM. I took some gross attempts at this with scancode-toolkit, but kept hitting OOM in attempting to scan the huge swaths within a VM.

My gross stab at this was something akin to:

sudo apt-get install -y python-dev libbz2 xzutils zlib1g libxml2-dev libxslt1-dev bzip2
wget https://github.com/nexB/scancode-toolkit/releases/download/v1.6.0/scancode-toolkit-1.6.0.tar.bz2
bzip2 -d scancode-toolkit-1.6.0.tar.bz2
tar xvf scancode-toolkit-1.6.0.tar
cd scancode-toolkit-1.6.0
./scancode --help
./scancode --verbose /usr

using an Ubuntu 16.04.1 VM as a base image.

It seems like this isn't the right tool for that sort of inventory/indexing of licenses by package, but I wanted to verify.

[pombredanne]

@heckj excellent question! and the short answer for now is not yet.

First thing: the OOM issue you have experienced should be all fixed in the develop branch... So you should try a checkout or download from develop HEAD instead, rather than the 1.6 release. We are a few days away to cut a version 2.0... so this is rather stable and much improved in so many ways.

Second: I reckon are your running ScanCode inside the as-installed VM, correct?

Then, ScanCode's goal is definitely to support this use case but we are not fully there yet:

1. Code to support Debian packages which is what matters to you here is in the work. We have not-yet-pushed code that handles .deb package archives and it will land in ScanCode at some time. The other part is the analysis of as-installed packages and here this would mean either:

• a static analysis of the Debian package database in a VM or container image which is a tad simpler as these are all text files.
• a proper dynamic invocation of apt commands in a running VM to obtain the list of installed packages and metadata.

2. For RPMs, .rpm archives support is available alright in the develop branch with the code in https://github.com/nexB/scancode-toolkit/blob/99f0a0de9062c44cc8cdb9cf3a7efd270dbdfd57/src/packagedcode/rpm.py but this is not handling yet as-installed RPMs. I have experimental support for Docker images support here with actual detection of as-installed RPMs in this tool here https://github.com/pombredanne/conan and this will eventually land in ScanCode soon. This is not a static analysis solution though yet as this requires running the Dcoker and rpm command in a copy of the containers to actually obtain the list of installed RPMs in each Dcoker layers and images. The proper solution for ScanCode would be a purely static analysis solution that would read the RPM database present in a VM image, container image or rootfs archive (e.g. read the berkeley DB or sqlite DB that contains the as-installed RPMs data) and not execute anything.

There is no tool that I know of that does any of this at the moment and no tool that I have heard of that would have such a goal. There are some tools focused on Docker image inventories (such as CoreOS clair) but they work only on containers and not VMs and rely to actually run the containers to execute an apt or rpm call. So if this is important for you, help is much welcomed to finish this if you have some appetite for it.

With that said, one of the thing that ScanCode will do alright now with a little bit of your help could be:

1. you run the appropriate rpm or apt commands to fetch the source and binaries for the installed packages in a VM and save them in a directory
2. you run the extractcode command on the directory to first extract the archives
3. you run scancode --license --copyright --info --package on this directory (eventually with the -n x option to run on x multiple processes in parallel.

[pombredanne]

We now have full (and I shall say 1st class support)to collect installed Debian packages data. The code lives here bur is best used from scancode.io with the rootfs or the docker pipelines.
Next will be RPM support

[pombredanne]

This is enabled by https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/debian.py#L79 and by the debut library https://github.com/nexB/debut

[pombredanne]

Now to best leverage this you should give a shot to https://github.com/nexB/scancode.io

[pombredanne]

@heckj I am closing this now as https://github.com/nexB/scancode.io/ (through scancode-toolkit and container-inspector and debian-inspector and a few other libraries) provides a comprehensive and extensive support to collect installed system installed packages for RPM distros (RHEL/Fedora/CentOS, Suse. Photon in progress), Debian distros (Debian and Ubuntu, distroless in progress) and Alpine. This is done in the context of Docker images or VM root filesystems.

For VMs the approach is NOT to run inside the VM, but to perform a static analysis on the VM images, and though technically it might be able to run inside the VM it does analyse I would not recommend it. In fact one of the unique feature of scancode.io is its ability to detect things WITHOUT running them (e.g. no docker run, no FS mount, no VM spin)
Feedback there is mucho welcomed!

[pombredanne]

FWIW, this feature has been significantly reworked in the latest develop branch here and will be enabled on ScanCode.io when ScanCode toolkit is released.