Install via pip tries to use cache directory in /usr/local/lib

[dengste]

I installed 'scancode-toolkit' via 'pip' on a Debian 9 box. On startup I get the following backtrace because a normal user obviously does not have permission to create files in /usr/local/lib:

 Traceback (most recent call last):
   File "/usr/local/bin/scancode", line 11, in <module>
 sys.exit(scancode())
   File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
 return self.main(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/scancode/utils.py", line 76, in main
 standalone_mode=standalone_mode, **extra)
   File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
 rv = self.invoke(ctx)
   File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
 return ctx.invoke(self.callback, **ctx.params)
   File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
 return callback(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
 return f(get_current_context(), *args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/scancode/cli.py", line 386, in scancode
 scans_cache_class = get_scans_cache_class()
   File "/usr/local/lib/python2.7/dist-packages/scancode/cache.py", line 93, in get_scans_cache_class
 cache_dir = fileutils.get_temp_dir(unicode(cache_dir), prefix=unicode(timeutils.time2tstamp()) + u'-')
   File "/usr/local/lib/python2.7/dist-packages/commoncode/fileutils.py", line 128, in get_temp_dir
 return tempfile.mkdtemp(prefix=prefix, dir=base)
   File "/usr/lib/python2.7/tempfile.py", line 339, in mkdtemp
 _os.mkdir(file, 0700)
 OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/.cache/scan_results_caches/2017-07-13T121227

[JonoYang]

@dengste You can opt to install pip packages to your home directory using the --user option. If you wanted to install it systemwide, you'd need to run pip with root access.

[pombredanne]

@JonoYang I think this has been installed system wide with sudo, right @dengste ?
I reckon this is a somewhat uncommon installation pattern that was never tried before!
What I think happens here is that ScanCode creates a cache for scan results and the license index. This cache is by default always created in the root directory which might end up being (incorrectly I reckon) /usr/local/lib/python2.7/.cache in the case of a global, system-wide installation... hence the permission denied error that you see.

A short term work around might be to do what @JonoYang suggests, e.g. a local user install. Not sure we have tested this either.

Another way is to install with pip in a local virtualenv: this is tested alright

Yet another way is fetch a release from https://github.com/nexB/scancode-toolkit/releases

[pombredanne]

Now this is a great bug ... especially since #487 absolutely needs this fixed.

The cache consists of two elements for now: a license index cache that could be global to a given installation and a transient per scan cache that would be best as local to a user.

For the sake of simplification, this might be best written to ~/.cache when the root directory of the scancode installation is not writeable (which is the case for a system install)....

There is another twist which is the license cache cannot be shared between multiple installations of ScanCode unless they have the exact same code and data. The license cache invalidation is done by taking a checksums of the installation directory tree per https://github.com/nexB/scancode-toolkit/blob/23d835ff2ad0587eedef4f9eebef87ee1ff0fe48/src/licensedcode/cache.py#L63 ... which breaks entirely when we have a pip installation, either local or global.

So it needs some thoughts.... one possibility could be to get the version of scancode (which is unique even when building/running from a checkout) and get or create a cache directory in ~/.cache/scancode/<version>/ at startup.... this would end in each user directory.
This cache would then be used for this user's runs both for the scan-level and and the license-level caches.

The whole license cache invalidation and check needs to be reworked too as in a pip install there is no root directory any more: the root ends up being two levels above a module directory hence it lands in the very ugly /usr/local/lib/python2.7/ directory.

@dengste Thanks ++ for this great find!

[pombredanne]

This is also related to #357

[pombredanne]

So it needs some thoughts.... one possibility could be to get the version of scancode (which is unique even when building/running from a checkout) and get or create a cache directory in ~/.cache/scancode// at startup.... this would end in each user directory.
This cache would then be used for this user's runs both for the scan-level and and the license-level caches.

Another thing which is about convenience for development or for a tarball/zip release installation: ideally in this case the .cache would be best local to the installation or checkout directory

[dengste]

Quick followup: Yes, I ran pip as root and did a systemwide install so that all users on the system can use it. When running as normal user, I would indeed expect scancode to use a directory in the user's home, either ~/.cache, or following the xdg specification (.local/share/scancode or similar).

[pombredanne]

@dengste Thanks. I guess ScanCode can run on a non-desktop so may be using a ~/.cache might be cleaner than relying on the slightly more complex XDG spec and environment (especially since its environment variables may not be present if nothing of X/Freedesktop is installed) ... It also allows to have consistent approach across OSes and seems common enough and clear.

[pombredanne]

this is eventually a problem with Docker images too, such as this: https://github.com/heremaps/oss-review-toolkit/blob/7b6fdfab6a35e4c1ce1362fd62e31ed023ef109d/Dockerfile

[pombredanne]

@sschuberth ping, for reference. This may impact your code .