#267 Remove memory leak from license detection - #376
Conversation
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* the bitmap-based one is used instead Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* Do not carry line_by_pos mapping in matches. Instead only inject start and end line in matches at the end using the query line_by_pos mapping kept only globally and not on a per-match basis. This was the source of a major memory leak when license matches were being updated and combined during a merge() run. * Use a list-based mapping for line_by_pos instead of a dic for smaller memory footprint. * Use slots for QueryRun attributes for smaller memory footprint * Replace "solid" attribute for Rules with a minimum_score that a match to a rule must equal or exceed. solid is now minimum_score: 100. Use this in other rules with various minimu score as needed. * Remove remaining references to "gaps". * Fix incorrect Rule thresholds computation for minimum lengths. * Do not use license matches cache for now (the sqlite-backed diskcache-based implementation is the source of major slowdown). * Various minor cleanup and updates on rules, licenses and their corresponding tests. * New batch of frequent tokens for license detection. * Add new tests contributed by @yahalom5776 * Add new license match filter for matches to a whole rule made of a single token that is surrounded by unknown or single letter tokens such as in "a b c d e GPL 1 2 3 4" to discard some false positive (in this case for a GPL). This required to add tracking of query tokens made on a single character. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
* New scan caching implementation using simple JSON files storage instead of a sqlite-backed storage. Since we have little or no contention, the strong ACID and locking offered by sqlite were slowing things down significantly by saturating disk I/Os. The process of caching scans is a write once, read once for each scanned file and therefore locking and atomic storage is not needed. * Also improve scan errors reporting. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
@sschuberth @yahalom5776 This is passing all checks and there quite a few changes I would like to build future code on. So barring a negative feedback, I will merge this over the week-end at the latest. |
|
Even if I did not have the time to review the code yet, I did give it a try by running it on one of our projects. And there is good news: One project that was previously causing the scanner to hang (I aborted the scan after 19 hours) completed after 30 minutes with this branch merged to the current |
|
@sschuberth you wrote:
Good news! I am not entirely surprised. I removed the caching of license matches and the caching of scans has been entirely refactored to use plain JSON files instead of a sqlite db (through diskcache). Both were using sqlite and sqlite was a major source of slowdown and was making the scan super I/O intensive for no reason. On a single process, I saw that this was at least about 4 times faster than before on my laptop. On multiple processes I achieved up to 10 files/sec for a license+copyright and 200 files/sec for just info. So one million files should take about 28 hours now and possibly much less on a faster-than-laptop-with-ssd server. |
|
We should create some benchmarks to watch for any perf changes btw. |
|
@pombredanne looks good so far! Thank you. |
That would be nice to have indeed, i.e. to have tests to identify performance regressions (or improvements). As a first step, looking at the times Travis CI builds take could be useful. Of course, this includes the setup and build times while we're only interested in the time to run tests, and the tests might be too many small projects whereas scanning performance probably is better measured with a single big project, but still. So I was quickly giving http://scribu.github.io/travis-stats/#nexB/scancode-toolkit a look, but unfortunately it does not render properly for me. See scribu/travis-stats#12. |
|
Thank you for the feedback. I think this is good enough for me to merge in now! |
... and other refinements and fixes detailed in the commit messages.
This should address all remaining issues in #267