segmentation fault error while running scans in an ubuntu instance in AWS

[balusarakesh]

I cloned into the github repo scancode-toolkit in an Ubnuntu instance (AWS-EC2 AMI - ami-06116566) and used the command ./scancode --help to configure scancode for the first time.
The next time when I try to run a command ./scancode --license file.txt I'm getting the error

/scancode: line 16:  8253 Segmentation fault      (core dumped) $SCANCODE_ROOT_DIR/bin/scancode "$@"

[pombredanne]

Do you have some more details on the OS version, architecture and version of Python used there?

[balusarakesh]

Ubuntu Server 14.04 LTS (HVM)

[balusarakesh]

@pombredanne Python version is 2.7.6

[pombredanne]

And which architecture? 32 or 64 bits? I also the exact Python version you get when you python

[balusarakesh]

64 bit

[balusarakesh]

2.7 version of Python

[yahalom5776]

Sorry to bring this up again but I seem to have a very similar issue when scanning a large project:
./scancode: line 16: 2977 Killed $SCANCODE_ROOT_DIR/bin/scancode "$@"

It is the current development branch installed on Debian 8 running in a VM with 2GB of RAM.

Python:
python --version: Python 2.7.9

OS:
uname -a: Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux

Let me know if you need more information. Thank you.

[pombredanne]

@yahalom5776 Thanks for the report. Can you tell me which command line options you used?

My best guess is that you have been prockilled' rather than segfaulted' based on your error message. This is something that should go away pretty soon.
The issue is simple: at the moment the results of scanning each file is accumulated in RAM for all files.
Eventually when scanning larger codebase, this will exhaust available RAM and the kernel may prockill the scancode process in this case. The solution is going to be saving on disk the scan results of each file on the go and then streaming the final JSON at the end such that the scan results are never fully held in memory.
This is something I am working on in the orm-models branch https://github.com/nexB/scancode-toolkit/tree/orm-models

The short term solution is to scan smaller chunks of a larger codebase. I am hoping to get some fix sometimes this week. In any case I will ping you to test the fix ass soon as this is ready.

[yahalom5776]

I used only the "-l" switch in combination with the HTML output because I wanted to get a quick overview of the codebase.

Thank you for the explanation, I am looking forward to the update. Please let me know if I can help with additional information or testing.

[sschuberth]

Just chiming in that I also see

scancode: line 16: 10193 Killed                  $SCANCODE_ROOT_DIR/bin/scancode "$@"

when running on Jenkins inside a VM and scanning Apache Flink (incl. the source code of all its Maven dependencies). The command line used simply is

scancode -f html . flink-scancode-50eaeec.html

I am hoping to get some fix sometimes this week.

Is there an update? Any work on this would be very much appreciated!

[pombredanne]

@sschuberth I am working on it and this is top priority

[pombredanne]

No ETA yet though

[JonoYang]

@sschuberth I am attempting to reproduce the issue. Could you tell me how to get the source code for Flink's Maven dependencies?