Limit number of urls and emails reported in a single file

[pombredanne]

At times some file -such as a test file- may contain a large number of emails and/or urls. Millions of them. We should limit the number of returned items per file in these cases to a sensible number like a few 100's and report a warning that not everything was scanned...

[sam09]

Anyone working on this? I would love to give it a try

[pombredanne]

@sam09 Go for it . Thanks!

[sam09]

I think we could probably limit the list of emails and urls to a 1000?
The get_urls and get_emails method can be modified to limit the number.
https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/api.py.

[pombredanne]

@sam09 anything with more than a 100 emails or urls is likely a data file with a huge amount of these. Actually 50 would already be a huge lot.

One consideration is how to report such cases of partial detection? some warning in the scan errors? some what to state like: 123673 URLs were detected in this file. Only the first 100 are reported... TBD. Suggestions are welcomed.

Also it could make sense to make the max number a CLI option and have a default to 50 or 100 for instance. In which case the issue of reporting partial detection may be a lesser issue.

[pombredanne]

and also since both urls and email would need a limit, having a single wrapper function that works for both would the way to go imho.... e.g. some generic function that accepts an iterable and yields a truncated iterable. It may even exists in the stdlib itertools.

[pombredanne]

e.g. https://docs.python.org/2.7/library/itertools.html?highlight=itertools#itertools.islice

[pombredanne]

so with this and a functools.partial you can create simple wrappers and use at first a hardcoded limit of say 50, and later a CLI option for the limit, where using 0 would mean return all.