--ignore plugin does not work on latest develop

[steven-esser]

[sesser@nexb ~] scancode -i --ignore=node_modules Code2/ --json-pp ~/here.json
Setup plugins...
Collect file inventory...
`ERROR: failed to run pre-scan plugin: ignore:
Traceback (most recent call last):
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/cli.py", line 870, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/plugin_ignore.py", line 85, in process_codebase
    remove_resource(resource)
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/resource.py", line 753, in remove_resource
    for descendant in resource.walk(self, topdown=False):
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/resource.py", line 1117, in walk
    for child in self.children(codebase):
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/resource.py", line 1141, in children
    return sorted((get_resource(rid) for rid in self.children_rids), key=_sorter)
  File "/Users/sesser/Code/scancode-toolkit/src/scancode/resource.py", line 1139, in <lambda>
    _sorter = lambda r: (r.has_children(), r.name.lower(), r.name)
AttributeError: 'NoneType' object has no attribute 'has_children'

Scan files for: info with 1 process(es)...
[------------------#-] 244 Scanned: pdfmake-0.1.18.LICENSE 

The --ignore plugin seems to be broken in some way. The above error is display and the scan still starts, but nothing gets ignored.

[JonoYang]

@MaJuRG What are you scanning in the Code2 directory?

[steven-esser]

@JonoYang 2 seperate directories with their own node_modules folders.

Ultimately, how do I structure my ignore to avoid scanning ANY node_modules directories that may appear at any different level of a codebase if I were to point at scancode at some top level root like a ~/code/ directory?

[JonoYang]

@MaJuRG The ignore plugin seems to work correctly when using the option --max-in-memory 0. I suspect that in the case where we do not store the entire codebase in memory, ScanCode does not load the parts of the codebase that are missing in memory before attempting to remove ignored resources.

[JonoYang]

@MaJuRG The issue is in the way we get resources from a codebase.
https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/resource.py#L634

We return None in the case that a resource id is not available before attempting to get and return the resource from the disk cache.

I have fixed this in a new branch:
https://github.com/nexB/scancode-toolkit/blob/956f14aecb4e66841474657a0aec3d4f56dca23b/src/scancode/resource.py#L634

[pombredanne]

@JonoYang soulns like you have a fix? that would be great.

[JonoYang]

@pombredanne I think I have a solution (https://github.com/nexB/scancode-toolkit/compare/1362-resource-not-loaded-before-removal), but I'm not sure how simply and consistently trigger the error so I can create a test for it.