MemoryError when scanning 100 MiB single-line JSON file

[sschuberth]

Slightly related to #616 (which is about the json2csv tool), I also see an error when scanning a (close to) 100 MiB JSON file with ScanCode version 2.2.1.post277.4d68f9377, on both Linux and Windows. I'm wondering whether ScanCode is still trying to interpret the JSON instead of handling it as a BLOB despite my not specifying the --package option, and whether that's causing the huge memory usage.

My full command line is

scancode --copyright --license --license-text --info --only-findings --strip-root --timeout 300 --processes 15 /dir/to/scan --json-pp result.json

Answering @pombredanne's question from #616:

JSON should not be treated as plain data when this is compact, single line.
Is this file a single line, like a huge js.map map file?

Yes, it is.

[sschuberth]

@pombredanne, judging from your question, I guess it would help to store the JSON file pretty-printed (to reduce the line length)?

[pombredanne]

Yes, exactly, I am adding pre-processors to deal with large JS and minified js and js map files to ensure that they if there is no line, they are formatted before they are scanned

[sschuberth]

But, like I think it's also mentioned in the other issue, isn't the better (because more generic) fix to stream the input? Or use something like memory mapped files?

[pombredanne]

@sschuberth quite likely: I am deep down in various solutions ATM 😸

[sschuberth]

Hey, this is how Gmail renders your emoticon:

Wouldn't that make a good purl logo? 😉

[pombredanne]

💃

[pombredanne]

emojis have gone a little too wild for my taste lately

[sschuberth]

For the record, I just realized this also happens for a 15 MiB CSV file in the same repos. That file's longest line is only 278 chars, so I'm a bit surprised to see the MemoryError also for that file. Or can that be a follow-up error of the first MemoryError in the JSON file?

[pombredanne]

@sschuberth I pushed to develop a set of enhancements. Can you try it on this?
I will add the same processing for CVSs BTW.
is this a public repo?

[sschuberth]

Thanks, I'll try to see if I can reliably reproduce it locally. And no, the repos / files are not public, unfortunately.

[pombredanne]

FWIW, I came to the conclusion that js.map files are really "archive-like" e.g. when they contain the full sources of all files, and they could be treated as an archive composed of each file in sourceContents: that would break things down nicely in palatable chunks (which is a strategy already used for patch files BTW). That will be a future refinement.
There are crazy big map files created these days packaging 100+ of npms in a single JS :|

[pombredanne]

re:

For the record, I just realized this also happens for a 15 MiB CSV file in the same repos. That file's longest line is only 278 chars, so I'm a bit surprised to see the MemoryError also for that file.

well the thing with CSV is that even with sort lines a whole CSV could be selected as a single unit for a copyright statement: say for instance that you have 5 columns and one of them contains a date: every line would be selected together as a single candidate for copyright detection because of this.
The solution would be to only ever consider one line at a time in these CSV cases I guess, as a statement or license would not ever spill from one line to the other; or actually may be not even spill from one cell to the other.

Or can that be a follow-up error of the first MemoryError in the JSON file?

That could not be the case IMHO