develop branch: scancode dies after scanning large number of files

[55jr55]

running on MacOS against repository with approx 30k files. Error messages below.

[-----------------------------------#] 29688 Scanning: Makefile [--------------------------#---------] 29689 Scanning: MANIFEST
Scanning done.
Traceback (most recent call last):
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/bin/scancode", line 9, in
load_entry_point('scancode-toolkit==1.6.3', 'console_scripts', 'scancode')()
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/lib/python2.7/site-packages/click/core.py", line 664, in call
return self.main(_args, *_kwargs)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/scancode/utils.py", line 61, in main
standalone_mode=standalone_mode, *_extra)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(_args, *_kwargs)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/scancode/cli.py", line 248, in scancode
results = scan(input, copyright, license, package, email, url, info, verbose, quiet)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/scancode/cli.py", line 315, in scan
scan_result.update(scan_one(res, scanners))
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/scancode/cli.py", line 332, in scan_one
scan = scan_func(input_file)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/scancode/api.py", line 178, in get_package_infos
package = recognize_packaged_archives(location)
File "/Users/jreynolds/git/nexB-GitHub-scanner/scancode-toolkit/src/packagedcode/recognize.py", line 58, in recognize_packaged_archives
return package.get_package(location)
AttributeError: type object 'CpanModule' has no attribute 'get_package'
LWRVLC02RD20TG8WL:tigo-source jreynolds$

[pombredanne]

Thank for reporting this . This is a bug.
I reckon you are using the develop branch, right?
This should not happen in the master branch. but this is a bug alright even in develop

[pombredanne]

@55jr55 can you tell which exact version of MacOSX you run?

[55jr55]

You are correct. I was using the develop branch. I'm on Mac OS 10.11.6.

[pombredanne]

@55jr55 that latest commit should fix the problem for now.

There is still something I do not like too much about the class hierarchy and code structure but develop should no longer exhibit this bug
I also merged in first @balusarakesh #281 which had most of the fix.
Note that develop is not always 100% stable... but your testing there is super valuable!

Unrelated since you are a Mac user in #290 I am thinking about dropping entirely support for older MacOSX versions < 10.9. In your experience would you see this as a big deal?