Add new fingerprint plugin by arnav-mandal1234 · Pull Request #1576 · aboutcode-org/scancode-toolkit

arnav-mandal1234 · 2019-05-30T19:01:51Z

Signed-off-by: arnav-mandal1234 arnav.mandal1234@gmail.com

Signed-off-by: Li <li@nexb.com>

codecov · 2019-05-31T06:02:14Z

Codecov Report

Merging #1576 into develop will decrease coverage by 2.05%.
The diff coverage is 43.07%.

@@            Coverage Diff             @@
##           develop   #1576      +/-   ##
==========================================
- Coverage    84.35%   82.3%   -2.06%     
==========================================
  Files          124     126       +2     
  Lines        14594   14659      +65     
==========================================
- Hits         12311   12065     -246     
- Misses        2283    2594     +311

Impacted Files	Coverage Δ
src/commoncode/fingerprint.py	`19.51% <19.51%> (ø)`
src/scancode/api.py	`72.78% <50%> (-22.32%)`	⬇️
src/cluecode/plugin_fingerprint.py	`90% <90%> (ø)`
src/scancode/extract_cli.py	`32% <0%> (-56%)`	⬇️
src/scancode/plugin_only_findings.py	`52% <0%> (-48%)`	⬇️
src/scancode/plugin_ignore.py	`30.9% <0%> (-45.46%)`	⬇️
src/scancode/plugin_mark_source.py	`56.25% <0%> (-40.63%)`	⬇️
src/scancode/interrupt.py	`32.89% <0%> (-11.85%)`	⬇️
src/scancode/resource.py	`75.14% <0%> (-10.84%)`	⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40e7192...cfcd73f. Read the comment docs.

codecov · 2019-05-31T06:02:14Z

Codecov Report

Merging #1576 into develop will decrease coverage by 0.53%.
The diff coverage is 70.73%.

@@             Coverage Diff             @@
##           develop    #1576      +/-   ##
===========================================
- Coverage     81.7%   81.16%   -0.54%     
===========================================
  Files          126      125       -1     
  Lines        15333    15478     +145     
===========================================
+ Hits         12528    12563      +35     
- Misses        2805     2915     +110

Impacted Files	Coverage Δ
src/cluecode/copyrights.py	`92.37% <ø> (-0.28%)`	⬇️
src/formattedcode/output_spdx.py	`89.67% <100%> (-4.25%)`	⬇️
src/commoncode/command.py	`81.1% <58.82%> (-5.34%)`	⬇️
src/extractcode/sevenzip.py	`82.64% <60%> (-0.23%)`	⬇️
src/commoncode/fileutils.py	`79.55% <73.07%> (-3.42%)`	⬇️
src/commoncode/codec.py	`74.35% <73.33%> (-21.8%)`	⬇️
src/formattedcode/output_html.py	`87.26% <0%> (-7.65%)`	⬇️
src/licensedcode/match.py	`79.15% <0%> (-4.85%)`	⬇️
src/textcode/markup.py	`93.47% <0%> (-4.35%)`	⬇️
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b64c93...3ec7854. Read the comment docs.

Signed-off-by: Li <li@nexb.com>

steven-esser · 2019-06-10T16:26:51Z

+    options = [
+        CommandLineOption(('-f', '--fingerprint'),
+                          is_flag=True, default=False,
+                          help='Scan <input> to generate fingerprint.',


We might want to be a bit more descriptive here and state why kind(s) of fingerprints the -f flag will generate.

@MaJuRG Okay. Since it is a very descriptive answer to your question, it would not be the right place to keep it IMHO. Can you please suggest me how should I frame it?

Sure. It does not have to be in-depth, but even just adding the name of the fingerprint or something like ... to generate simhash fingerprints for use in similarity matching. Something like that.

Sure. will change it.

steven-esser · 2019-06-10T16:27:41Z

+import hashlib
+
+hash_length = 128
+shingle_length = 3


Standard convention for constants like these should be all caps: HASH_LENGTH, SHINGLE_LENGTH to distinguish them from other variables.

Sure. Will change it.

steven-esser · 2019-06-10T17:58:05Z

+    {
+      "path": "factorial-test.java",
+      "type": "file",
+      "fingerprint": "00000001110011111011010000110111101011001110110100111111011111011000111001000001100001001111010011011000011001110011000100100111",


is there another way to store this fingerprint, or does it have to be a binary string? The length of these fingerprints is unsightly IMO.

@arnav-mandal1234 Store the fingerprint as a hex string instead of a binary string

@MaJuRG @JonoYang We can keep the fingerprint in hex but then we have to shift most of the code to the delta code repo and do the computation there. Would it be the right thing to do it?

We just want to calcuate the fingerprints for each file in scancode-toolkit. When it comes to the actual comparison aspect, that will be handled in deltacode (calculating the hamming distance, etc). I do not think we have to move any code over, correct? The additions you made were simply to:

create the fingerprint for each file

add it to scan output

When we go to actually check two separate scan output files, deltacode will be doing those comparisons. Does this make sense?

@MaJuRG Yes. That is what in the back of my mind too.
Sorry I misunderstood @JonoYang 's comment. So, basically you want these fingerprints to be in hex. then after we get these hex string on the deltacode side, we again convert them to binary and calculate hamming distance. Right?

steven-esser · 2019-06-10T17:59:08Z

+    def test_generate_fingerprint1(self):
+        test_file = self.get_test_loc('fingerprint/fingerprint-test1.java')
+        result = generate_fingerprint(test_file)
+        assert result == '01001110010000101101100011000000111011010110011010010011011001010100100001100110010000100101010001010001001000010000010000010111'


See comment above. Is there another way to store this fingerprint?

steven-esser · 2019-06-10T18:00:18Z

@pombredanne @JonoYang Your review would be welcome as well.

JonoYang · 2019-06-10T18:02:40Z

+    weighted_list = get_weightedlist(token_list)
+    fingerprint = process_weightedlist(weighted_list)
+
+    return "".join(str(bit) for bit in fingerprint)


Style note: we use single quotes ' when declaring strings.

@JonoYang sure will change this.

JonoYang · 2019-06-10T18:05:58Z

+    {
+      "path": "factorial-test.java",
+      "type": "file",
+      "fingerprint": "00000001110011111011010000110111101011001110110100111111011111011000111001000001100001001111010011011000011001110011000100100111",


@arnav-mandal1234 Store the fingerprint as a hex string instead of a binary string

Signed-off-by: Li <li@nexb.com>

pombredanne

A few general questions: do we want this as a built-in plugin that is always available? or is this an external plugin?

Also I am sure that cluecode is not the right module for this: until now this was about copyrights and "greps" only.

I made several comments inline too.

pombredanne · 2019-06-11T13:39:36Z

@@ -0,0 +1,60 @@
+#
+# Copyright (c) 2018 nexB Inc. and others. All rights reserved.


We are in 2019 alright. But having no year is probably best

Sure will change this.

pombredanne · 2019-06-11T13:41:06Z

+        return fingerprint
+
+    def get_scanner(self, **kwargs):
+        from scancode.api import get_fingerprint


The scancode.api module is a legacy wart. Avoid using this for this code. Instead put it all in this same module here.

@pombredanne should I shift it to the commoncode/fingerprint.py?

pombredanne · 2019-06-11T13:41:52Z

+
+def generate_fingerprint(location):
+    """
+        Return fingerprint of the file at `location`.


Theer 4 extra spaces to remove in from the Return

Sure will change this.

pombredanne · 2019-06-11T13:43:31Z

+    """
+        Return fingerprint of the file at `location`.
+    """
+    token_list = get_tokenlist(location)


why do you qualify each variable and functions with a type?
You should have instead tokens = get_tokens(location) here and in other similar places below and elsewhere

Okay. will change all the similar variable and function names.

pombredanne · 2019-06-11T13:44:14Z

+
+def get_tokenlist(location):
+    """
+       Return a list of tokens for the file at `location`.


Remove 4 extra spaces.

Okay. Will do that.

pombredanne · 2019-06-11T13:56:48Z


 from commoncode.filetype import get_last_modified_date
 from commoncode.hash import multi_checksums
+from commoncode.fingerprint import generate_fingerprint


The import needs to be properly ordered

In sorted order?

Yes sorted alphabetically

Okay. will change it.

pombredanne · 2019-06-11T13:57:20Z

    result['is_script'] = bool(collector.is_script)
    return result

+def get_fingerprint(location, **kwargs):


Do not put this code in this module. Keep it with your plugin instead

Sure will keep this in commoncode/fingerprint.py

pombredanne · 2019-06-11T13:58:23Z

@@ -0,0 +1,15 @@
+public class Factorial


Can you include real code and text and data files as tests rather than only made up ones?

Sorry I didn't get you. Please explain. :)

He means you should use actual source found in the wild, like code from an open source repository.

This is a better representation of files scancode will actually come across in the wild.

There should also be a verity of these, differing in lines of code AND language.

Okay. I will change it then. :)

pombredanne · 2019-06-11T13:59:31Z

@@ -0,0 +1,114 @@
+#


Why is this under commoncode when your plugin is in cluecode?

Actually, all the main functions are in commoncode i.e. fingerprint.py. And the plugin is in cluecode. what should be the structure? should I move everything to commoncode?

Signed-off-by: Li <li@nexb.com>

…#253 Signed-off-by: Li <li@nexb.com>

Signed-off-by: Li <li@nexb.com>

Unicode string literals are simply converted into string literals, which are always Unicode in Python 3. Unicode raw strings (in which Python does not auto-escape backslashes) are converted to raw strings. In Python 3, raw strings are always Unicode. Signed-off-by: Abhishek Kumar <abhishek.kasyap09@gmail.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

range() is now its own generator-like class in Python 3 and was a list in Python2. We now handle what is a generator-like object properly on both versions. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

For now run only the commoncode tests on Python 2.7 to tune the setup Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

These are in the form of: First Last <email> (c) year Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add new License rules and licenses

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

…rithm Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

…4/scancode-toolkit into gsoc_fingerprint

pombredanne · 2019-07-23T04:55:49Z

@arnav-mandal1234 there seems to be something rather wrong with this PR and its 6710 modified files! is there some issues with your rebasing?

arnav-mandal1234 · 2019-07-23T05:37:46Z

@arnav-mandal1234 there seems to be something rather wrong with this PR and its 6710 modified files! is there some issues with your rebasing?

@pombredanne Something went wrong. I had opened a new PR with the same changes and it got merged. :)
You can close this PR or if you want I can do that.

steven-esser · 2019-07-23T06:05:17Z

@pombredanne This can be closed. #1651 was merged and was the replacement for this.

Li added 2 commits May 27, 2019 21:23

Include pkginfo lib #253

6eb931d

Signed-off-by: Li <li@nexb.com>

Include pkginfo lib and add the parser #253

e4f62b7

Signed-off-by: Li <li@nexb.com>

steven-esser reviewed May 30, 2019

View reviewed changes

Comment thread src/commoncode/fingerprint.py Outdated

Li added 5 commits June 2, 2019 03:54

Add the function and test to parse wheel file #253

794a684

Signed-off-by: Li <li@nexb.com>

Add new parser2 function for pypi #253

d7ebe5a

Signed-off-by: Li <li@nexb.com>

Add tests for the parse function #253

04c74ae

Signed-off-by: Li <li@nexb.com>

Add the dparse lib #253

fe2f4f4

Signed-off-by: Li <li@nexb.com>

Include dparse in the dependency #253

0f4bdae

Signed-off-by: Li <li@nexb.com>

steven-esser suggested changes Jun 10, 2019

View reviewed changes

steven-esser assigned pombredanne, JonoYang and steven-esser Jun 10, 2019

JonoYang reviewed Jun 10, 2019

View reviewed changes

Add parse with dparse #253

c64548e

Signed-off-by: Li <li@nexb.com>

pombredanne changed the title ~~Adds new fingerprint plugin~~ Add new fingerprint plugin Jun 11, 2019

pombredanne requested changes Jun 11, 2019

View reviewed changes

Li added 3 commits June 11, 2019 23:08

Add test for dparse function #253

0c579ad

Signed-off-by: Li <li@nexb.com>

Incldue the function to parse the dependencies and set back to package …

30ed9dc

…#253 Signed-off-by: Li <li@nexb.com>

Include the requirement/version info #253

e0d9ba7

Signed-off-by: Li <li@nexb.com>

JonoYang reviewed Jun 14, 2019

View reviewed changes

Comment thread src/cluecode/plugin_fingerprint.py

Comment thread src/commoncode/fingerprint.py Outdated

JonoYang reviewed Jun 18, 2019

View reviewed changes

Comment thread src/scancode/fingerprint.py Outdated

Comment thread src/scancode/fingerprint.py Outdated

JonoYang reviewed Jun 18, 2019

View reviewed changes

Comment thread src/scancode/fingerprint.py Outdated

Comment thread tests/scancode/data/plugin_fingerprint/fingerprints.expected.json Outdated

JonoYang reviewed Jun 19, 2019

View reviewed changes

Comment thread src/scancode/fingerprint.py Outdated

Comment thread src/scancode/fingerprint.py Outdated

Abhishek-Dev09 and others added 4 commits June 20, 2019 13:00

Do not treat bytes as a string on Python3 #295

ffeef68

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Port functional.py to Python3 #295

5a2ac93

range() is now its own generator-like class in Python 3 and was a list in Python2. We now handle what is a generator-like object properly on both versions. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update Azure CI #295

7c576a0

For now run only the commoncode tests on Python 2.7 to tune the setup Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne and others added 25 commits July 11, 2019 16:15

Add misc new license rules

8f1bd80

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not truncate some copyrights

c7f3e27

These are in the form of: First Last <email> (c) year Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve copyright test failures output

688cf97

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve some copyright tests expectations

b728748

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not exit on errors if sourcing configure #1647

89a5af3

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge pull request #1646 from nexB/license-updates-07-10

d7fee0b

Add new License rules and licenses

adds plugin_fingerprint file and imports

a73adf5

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds plugin_fingerprint

650594c

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

adds fingerprint plugin in the setup.py

693ad77

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds functions and imports to incorporate fingerprint generation algo…

54956d0

…rithm Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds working fingerprint generations algorithm

6b00d66

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Modifies fingerprint.py to generalised code

089a081

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Fixes edge cases in fingerprint.py

3eda4b9

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

fingerprint.py code refactored

cd18cb3

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Modifies help.txt for adding new fingerprint plugin

0d4c3db

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds tests for plugin_fingerprint.py

3f7bc62

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds tests for fingerprint.py

298a7ee

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

code refactored

31f6bf3

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Changes bitarray_from_bytes() method to staticmethod

f54d388

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds similarity matching tests

8c8fc20

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Adds method docs in fingerprint.py

bf5bd63

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Refactors hamming_distance() function

7943fdb

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Rebased with master and refactored code structure

1ac4c1c

Signed-off-by: arnav-mandal1234 <arnav.mandal1234@gmail.com>

Merge branch 'gsoc_fingerprint' of https://github.com/arnav-mandal123…

fcf106a

…4/scancode-toolkit into gsoc_fingerprint

Refactors code structure

3ec7854

steven-esser closed this Jul 23, 2019

arnav-mandal1234 mentioned this pull request Aug 24, 2019

Add fingerprint plugin #1651

Merged

		@@ -0,0 +1,60 @@
		#
		# Copyright (c) 2018 nexB Inc. and others. All rights reserved.

Uh oh!

Uh oh!

Conversation

arnav-mandal1234 commented May 30, 2019

Uh oh!

Uh oh!

codecov Bot commented May 31, 2019

Codecov Report

Uh oh!

codecov Bot commented May 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steven-esser Jun 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steven-esser commented Jun 10, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codecov Bot commented May 31, 2019 •

edited

Loading

steven-esser Jun 11, 2019 •

edited

Loading