Long scan timeout for #267 - #369
Merged
Merged
Conversation
pombredanne
force-pushed
the
long-scan-timeout
branch
from
November 15, 2016 00:50
c313697 to
abc0cdc
Compare
Member
Author
|
The latest commits now make the memory check the default |
sschuberth
reviewed
Nov 15, 2016
| while True: | ||
| try: | ||
| result = scanned.next() | ||
| scan_sucess, scanned_rel_path = result |
Collaborator
There was a problem hiding this comment.
Minor typo: Should say "scan_success".
Collaborator
|
Looks good to me from a cursory view, thanks for working on this. On the CI side, the one Travis build seems to have timed out and probably just needs to be retriggered. For AppVeyor things look a bit worse, probably some tweaking of |
Member
Author
|
@sschuberth I fixed the typos and a few more. Thanks ++ for this review! I need to fix the CI failures now. |
pombredanne
force-pushed
the
long-scan-timeout
branch
2 times, most recently
from
November 17, 2016 00:40
5326369 to
03b6312
Compare
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* based on http://stackoverflow.com/questions/29494001/how-can-i-abort-a-task-in-a-multiprocessing-pool-after-a-timeout/29495039#29495039 * thanks to @dano for this, kindly also dedicated to public domain * this is use a thread and a timeout. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* handle error cases * add new get_scans_cache_class function to use a cache on a stable dir Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* start and end is no longer part of it. Instead users should provide their own start and end "displays". Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* a scan can be interrupted if it takes too long and exceeds a timeout * a scan can run on multiple processes with the -n option * --verbose output is always send to stdout. * a summary of the scan statistics is displayed at the end. * the tests and extractcode are not up to date/working yet Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* refine the events display and corresponding tests * use new utils.get_relative_path signature Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* Update tests to new utils.get_relative_path semantics * Add tests for multiprocessing and timeouts Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* it was always going to be the MAX_SCAN_TIMEOUT otherwise. * it must be the smallest of computed timeout and MAX_SCAN_TIMEOUT. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* until now only timeout errors were reported Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* if the SCANCODE_EXPERIMENTAL_MAX_MEMORY env var exists then the memory limit will be enforced. Otherwise only timeouts are enforced Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* Ensure that we check megabytes not gigabytes * Streamline code and remove duplicates * Add tests Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* timeout and max mem computaion have moved to interrupt.py * feature switch for max memory removed Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* from PR review by @sschuberth Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* were failing on CI * run only on two processes * increase max_memory and scan with --license to blow it Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* This is replacing globals. This should be more robust on Windows that
does not fork and may have been a cause of failing tests on CI.
* Also ensure that POSIX paths are always returned
* Also improve the reporting of scan errors
* Streamline code, ensure errors have a consistent structure
* Do not include full traceback by default instead use the
new --diag option to include full traceback only if requested
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* since some tests are not exceeding quoats now and then on CI we now test that at least one of the scans is being interrupted from a timeout or max_memory rather than excepting all to be interrupted. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* since some tests are not exceeding quoats now and then on CI we now test that at least one of the scans is being interrupted from a timeout or max_memory rather than excepting all to be interrupted. * Improve test assertion Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne
force-pushed
the
long-scan-timeout
branch
from
November 17, 2016 00:57
03b6312 to
d9020e3
Compare
* Only run two threads instead of three * Improve timeouts test with files guaranteed to exceed a test timeout to scan Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
pombredanne
force-pushed
the
long-scan-timeout
branch
from
November 17, 2016 06:46
daad7a9 to
e87e381
Compare
Contributor
|
And at least one of the builds passed on Mac too: https://travis-ci.org/nexB/scancode-toolkit/builds/176613254 |
Contributor
|
Linux and Windows seem OK |
Contributor
|
All tests pass now: 🍏 💚 |
Collaborator
|
Let's get it merged for some wider exposure! :-) |
Member
Author
|
@sschuberth agreed. Further fixes if needed can branch as needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #267 the key issue is some pathological, hard to diagnose scan that takes a long time and eventually uses a lot of RAM to get prockilled (at least on Linux).
This should both help handle the issue and help diagnose it:
a scan will be interrupted if it takes too long and exceeds a timeout: so if there is a problem somewhere, it should timeout rather than suck all the memory and get proc killed. The approach is to run each scan in a thread with a timeout using code from @dano
The timeout is computed for each file being scanned. It has a minimum of 60 seconds and a maximum of 600 seconds. Each file bigger than 1MB gets an extra 30 of timeout for each extra MB up to the max 600s.
a scan can run on multiple processes with the
-n or --processesoption (this is a nice side effect of handling timeouts and can also speed up scanning of course, but would use more RAM). When running on multiple process the memory used by each process may vary but would be about 600MB + size of the scanned file in memory + some extras.-
-verboseoutput is always send to stdout, as you requested by @sschuberth which will help to diagnose issues for background, CI-like scan jobs. So running a --verbose scan on this would likely tell exactly which is the file that is causing an error.a summary of the scan statistics is displayed at the end of a scan: this helps diagnose errors as it does list all the paths that failed to scan alright or did timeout. It also provide some timing and scanning speed details.