Allow cache location configuration

[pombredanne]

Since this could be eventually large and does not play well with any network storage (NTFS shares or NFS) the location where the cache is stored on disk should be configurable, ideally with env variables

[pombredanne]

Here is my proposal for this:

• For development usage (from a git checkout) we would detect the presence of .git directory and use a .cache directory (or may be a .tmp directory) at the root of the checkout as today.

• For non-development installations (e.g. without a .git directory) we would by default store all the temporary and cached files in a .scancode dir in the user home directory. Since there can be multiple installations and versions installed at the same time, we would use a unique key for each installation as a sub-directory in the .scancode directory. That key will be computed based on a hash of the code tree the same way it is computed for the license cache validity check today in here https://github.com/nexB/scancode-toolkit/blob/c57dab57ff74723e783b7c0dfeb7032f9bb7e84e/src/licensedcode/cache.py#L61 . This would ensure that the sub directory is unique to an installation and that even if some files are updated manually in that installation, a new temp directory would be created.

• additionally if a SCANCODE_TEMP_DIR environment variable is defined pointing to a directory path, this would be used and override the location used for caching and temp files both in dev and regular mode.

Last we could also include some crude and quick stats on the amount temporary disk space used in the current temp dir.

In all cases, deleting the whole temp directory would have no impact (unless this is done while a scan is mid air of course)

Feedback welcomed!

[haikoschol]

@pombredanne Your proposal makes sense to me and I'd like to give implementing it a try (in order to fix #685). Just two questions/suggestions:

• Why look for a .git directory to detect development mode instead of SCANCODE_DEV_MODE?
• Wouldn't it be easier to use the same directory structure in both cases for temporary files (i.e. always use .cache/scancode/<unique_id>/...)?

[pombredanne]

@haikoschol excellent!

Why look for a .git directory to detect development mode instead of SCANCODE_DEV_MODE?

well, either way works: your call SCANCODE_DEV_MODE is a bit of a wart I created IMHO . I am not very happy about this

Wouldn't it be easier to use the same directory structure in both cases for temporary files (i.e. always use .cache/scancode/<unique_id>/...)?

That works too. Having it everywhere the same is indeed better: either .scancode/ or .cache/scancode everywhere is indeed best.

[haikoschol]

True, a tag file is not ideal but the concept of "dev mode" and "user mode" is very useful, I think. The advantage of a tag file is that it is explicit and easy to create for testing/reproducing bugs/etc.

[pombredanne]

so now you are convincing me that it may not be such a wart after all :P

[pombredanne]

See instead this #685 (comment) for a comprehensive analysis and suggested solution for both cache and temp files