Failed to configure on macOS

[pombredanne]

reported in Gitter by @susg

Hi all, I was setting up scancode toolkit and got the following error while using the command: source configure.

    Configuring ...
    Building license index...
    Traceback (most recent call last):
    File "/Users/sushant/scancode-toolkit/etc/conf/base.py", line 59, in <module>
    build_license_cache()
    File "/Users/sushant/scancode-toolkit/etc/conf/base.py", line 56, in build_license_cache
    cache.reindex()
    File "/Users/sushant/scancode-toolkit/src/licensedcode/cache.py", line 145, in get_or_build_index_through_cache
    rules_data_dir=_rules_data_dir)
    File "/Users/sushant/scancode-toolkit/src/licensedcode/models.py", line 453, in get_rules
    rules = list(load_rules(rules_data_dir=rules_data_dir))
    File "/Users/sushant/scancode-toolkit/src/licensedcode/models.py", line 550, in load_rules
    raise Exception(msg % locals())
    Exception: Orphaned files in rule directory: '/Users/sushant/scancode-toolkit/src/licensedcode/data/rules'
    /Users/sushant/scancode-toolkit/src/licensedcode/data/rules/.DS_Store

Failed to execute command:
/Users/sushant/scancode-toolkit/bin/python "/Users/sushant/scancode-toolkit/etc/conf/base.py". Aborting...

[susg]

In src/licensedcode/models.py if we change line 537 to:
if not (data_file.endswith('~') or data_file.endswith('.DS_Store')):, I guess it will solve the problem.

[pombredanne]

@susg that's an approach but we can do better and https://github.com/nexB/scancode-toolkit/blob/d53aac8c0b3e622117151312d535a90a52fc5e6e/src/licensedcode/models.py#L537 is a wart of mine IMHO that should be refined.

The directories are walked with file_iter() here https://github.com/nexB/scancode-toolkit/blob/d53aac8c0b3e622117151312d535a90a52fc5e6e/src/licensedcode/models.py#L514

file_iter() comes from https://github.com/nexB/scancode-toolkit/blob/d53aac8c0b3e622117151312d535a90a52fc5e6e/src/commoncode/fileutils.py#L435 and can accept things to ignore with an ignored callable arg. e.g. a callable accepting a location argument and returning True if the location should be ignored.

https://github.com/nexB/scancode-toolkit/blob/d53aac8c0b3e622117151312d535a90a52fc5e6e/src/commoncode/ignore.py#L39 provides some convenience and default to build ignored callables using the fileset matching capabilities in https://github.com/nexB/scancode-toolkit/blob/00a078e358036d06a3a361949799383599a0df5b/src/commoncode/fileset.py

The better approach would therefore to pass some ignores by building a callable with default ignores.

Here are two examples for getting such callable:

• https://github.com/nexB/scancode-toolkit/blob/4e63a6775a895fdec5a3a53f96013399fa34b1b7/src/licensedcode/cache.py#L56 builds a function _ignored_from_hash with a small set of custom ignores.

• https://github.com/nexB/scancode-toolkit/blob/6a7c1d8281b3e316176340947a8cdde12f05983a/src/scancode/cli.py#L817 builds a function ignorer from default VCS ignores and plugin-provided ignores https://github.com/nexB/scancode-toolkit/blob/6a7c1d8281b3e316176340947a8cdde12f05983a/src/scancode/cli.py#L823

In your case, the best way would be IMHO to use the defaults in default_ignores in https://github.com/nexB/scancode-toolkit/blob/develop/src/commoncode/ignore.py#L297 and likely adding the ~ swap editor files to the mix.

[susg]

https://github.com/nexB/scancode-toolkit/blob/d53aac8c0b3e622117151312d535a90a52fc5e6e/src/commoncode/ignore.py#L78
This line already includes the .DS_Store files in the default ignorers, so why is the exception being raised?

[pombredanne]

@susg sorry for the late reply... I had missed your message.
The reason why an exception is raised is that the code in ignore.py is not used in the licensedcode.models.py yet and should be used