json2csv - memory error when input is huge [need to support Wind64]

[chinyeungli]

Traceback (most recent call last):
  File "json2csv.py", line 256, in <module>
    cli()
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 644, in main
    rv = self.invoke(ctx)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "json2csv.py", line 252, in cli
    json_scan_to_csv(json_input, csv_output)
  File "json2csv.py", line 80, in json_scan_to_csv
    rows = list(flatten_scan(scan_results, headers))
  File "json2csv.py", line 132, in flatten_scan
    lic = OrderedDict()
MemoryError

I get the above MemoryError when dealing with a 127MB input JSON file.
However, @JonoYang ran the same input with no problem.

[steven-esser]

@chinyeungli How much RAM does your machine have?

[chinyeungli]

16GB

[JonoYang]

@chinyeungli

The version of Python shipped with ScanCode that is used by Windows is 32-bit, whereas the version of Python shipped with ScanCode that is used by Linux is 64-bit. I used the test from here to determine if a 32-bit or 64-bit version of Python is being used. I'm guessing that the MemoryError is due to the 32-bit version of Python not being able to use as much memory as the 64-bit version of Python.

In this case, I would suggest running json2csv.py in a POSIX environment as there is not much we can do to reduce memory usage when dealing with such a large scan file.

[pombredanne]

@JonoYang I think this can be improved with streaming and should not be closed.

[JonoYang]

@pombredanne Is there any particular reason why the 32-bit version of Python is used for ScanCode on Windows?

[pombredanne]

@JonoYang the only reason is that we did not port and test all the libraries that are native to Win64.

[sschuberth]

I'm now seeing a very similar error when scanning a 94 MiB JSON file with ScanCode version 2.2.1.post277.4d68f9377, on both Linux and Windows. I'm wondering whether ScanCode is still trying to interpret the JSON instead of handling it as a BLOB despite my not specifying the --package option, and whether that's causing the huge memory usage.

My full command line is

scancode --copyright --license --license-text --info --only-findings --strip-root --timeout 300 --processes 15 /dir/to/scan --json-pp result.json

[pombredanne]

@sschuberth ScanCode is unlikely to try to interpret it and this is likely the root of the issue.
JSON should not be treated as plain data when this is compact, single line.
Is this file a single line, like a huge js.map map file?

I am working on sorting out this kind of issues with both js.map map files and minified JS.

Note this is not a json2csv issue, but should be a new issue IMHO.

[sschuberth]

Note this is not a json2csv issue, but should be a new issue IMHO.

Agreed: #958.

[chinyeungli]

any update on this?
I have the same MemoryError when trying to run this script in Windows OS with a ~400M json file, but I have different error message this time:

Traceback (most recent call last):
  File "etc\scripts\json2csv.py", line 265, in <module>
    cli()
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\CYL\git\scancode-toolkit\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "etc\scripts\json2csv.py", line 261, in cli
    json_scan_to_csv(json_input, csv_output, prefix_path, include_text)
  File "etc\scripts\json2csv.py", line 73, in json_scan_to_csv
    scan_results = load_scan(json_input)
  File "etc\scripts\json2csv.py", line 52, in load_scan
    scan = jsonf.read()
  File "C:\Users\CYL\git\scancode-toolkit\lib\codecs.py", line 674, in read
    return self.reader.read(size)
  File "C:\Users\CYL\git\scancode-toolkit\lib\codecs.py", line 480, in read
    newchars, decodedbytes = self.decode(data, self.errors)
MemoryError