Transparent extraction of archives

[pombredanne]

As noted in #3, we do not extract and scan at the same time.

A better way would be to handle internally an archive as if it were a special type of directory (both contain files after all), and when a single archive scan is requested (or when archives are found in a larger scan) we could extract these temporarily to a temp directory, scan the extract and return the results. This would require a bit more thinking to get it right.
At a high level a tree with archives would be considered the same as a tree with directories. Archives would become just a special type directory-like containers for more files.

We could expose an os.walk-like function that would transparently extract archives to a temp directory and yield a real path and the temp location of a given file

[pombredanne]

I need some input on the best way transparent extraction would behave when reporting paths to files inside extracted archives and whether to handle extraction outside or inside/in-place of the scanned tree.

In the current way, extractcode needs to be launched separately prior to a scan and each archive is extracted inside the scanned tree in a new directory created side-by-side with its archive and named after the archive file name with an -extract suffix added.
The paths reported in the scan are both the archive path and separately the paths to the -extract directory and its contained files.

With a transparent archive extraction, we could either extract in place or not:

1. keep the same approach, where the extraction is just integrated in the scan and the archive physically extracted in-place as when running extractcode

2. during the scan, extract each archive to a temporary location instead (outside of the scanned tree).
   The corresponding extracted files would not exist in the scanned tree but instead extracted out of tree in temp files that would no longer be available once the scan is complete which may be either a good thing or a confusing thing.

With a transparent archive extraction, we could report a real or a "virtual" path to an extracted file in the scanned results:

1. Report path/to/archive.zip-extract/some/dir/and/file and also report path/to/archive.zip
2. Report path/to/archive.zip/some/dir/and/file without an -extract suffix as if an archive were a directory-like structure. And we would also report path/to/archive.zip. Here the path would be "virtual"

And if the extraction is not done in-place, then the paths would always be kinda virtual and absent of the scanned tree.

So this about either extracting in place or not and which path to report in the scan results.

@akintayo @chinyeungli @MaJuRG @nakami @rakeshbalusa @sschuberth @yahalom5776 Each of you has been involved with issues related to archive extraction. What would be your take and preference? What could be alternative ways?

Thanks for your input!

[sschuberth]

I believe extracting out-of-tree is the better / safer approach, simply because you don't have to worry about whether the directory you plan to extract to already exists. Also, it gives a cleaner separation between the "primary" source code / files, and files coming from the archives.

Consequently, regarding the reporting I do like path/to/archive.zip/some/dir/and/file better because with out-of-tree extraction there simply is not need to append an -extract suffix anymore.

But speaking of in-place vs. out-of-tree extraction I wonder whether creating files is necessary at all. Why not simply stream the files from the ZIP and directly pass their contents to the scanning engine? That would probably increase performance by reducing file I/O, and also get rid of the need to delete the temporarily extracted files afterwards.

[pombredanne]

@sschuberth your idea to stream read archives is intriguing. It can be done actually not only on zips but also on most archives that are handled by libarchive or in Python code. It would not work though for these handled by 7zip I thing. I will need to weight the benefits vs. the code simplicity.

[akaihola]

Some notes from the point of view of our use case:

• our packages are on a network drive
• currently extractcode only can extract next to the archives
• it doesn't make sense to extract onto the network drive for obvious reasons
  • maybe add an option for the destination root path?
• would prefer to not read and extract hundreds of packages from a network drive every time before scanning
• optimally, results would be cached
  • no extraction nor scanning would be done if results matching the package name+version and scancode version are already in the cache

What I'm using currently is our own wrapper script which extracts and runs scancode for each package separately, and stores results in per-package JSON files on a local drive. It also skips package files for which a JSON file already exists.