Add origin summary plugin by JonoYang · Pull Request #1588 · aboutcode-org/scancode-toolkit

JonoYang · 2019-06-07T21:50:30Z

This is a plugin that performs license and copyright holder summarization at the directory level. If a certain copyright holder and license expression shows up in 75% of files in a directory, then that copyright holder and license expression is brought to the directory level. The summarized clues are stored in a new field called origin_summary, which is a mapping that contains three keys: license_expression, holders, and count (the number of files with this particular license expression and copyright holders). There is another new field called summarized_to, which is a string that contains the path to the directory a Resource has been summarized to.

* If a license or copyright is detected in 80% of files in a given directory, that license and copyright is summarized at the directory level Signed-off-by: Jono Yang <jyang@nexb.com>

* Register origin summary plugin Signed-off-by: Jono Yang <jyang@nexb.com>

* Define majority to be 50% or more of a directory Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add new flags for summarized rows or if a row is a summary Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Group holders and licenses together when counting * Check to see if copyrights or licenses is an attribute instead of requiring the copyright or license option to run the plugin Signed-off-by: Jono Yang <jyang@nexb.com>

* Increase majority threshold to 0.75 Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* This is so we do not have to walk the codebase twice to find children that have been summarized * We can instead just iterate through the list of RIDs and modify them * Update tests due to previous methods of tagging summarized Resources being flawed Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

codecov · 2019-06-07T22:31:20Z

Codecov Report

Merging #1588 into develop will increase coverage by 0.07%.
The diff coverage is 93.33%.

@@             Coverage Diff             @@
##           develop    #1588      +/-   ##
===========================================
+ Coverage    81.22%   81.29%   +0.07%     
===========================================
  Files          126      126              
  Lines        15612    15655      +43     
===========================================
+ Hits         12681    12727      +46     
+ Misses        2931     2928       -3

Impacted Files	Coverage Δ
src/commoncode/ignore.py	`74.35% <ø> (ø)`	⬆️
src/packagedcode/rubygems.py	`66.55% <ø> (ø)`	⬆️
src/packagedcode/nuget.py	`78.04% <ø> (ø)`	⬆️
src/commoncode/fileutils.py	`79.61% <100%> (+0.06%)`	⬆️
src/packagedcode/models.py	`93% <100%> (+0.21%)`	⬆️
src/packagedcode/about.py	`71.92% <100%> (+1.02%)`	⬆️
src/packagedcode/chef.py	`94.39% <100%> (+0.1%)`	⬆️
src/packagedcode/npm.py	`86.54% <100%> (+0.12%)`	⬆️
src/packagedcode/pypi.py	`85.13% <100%> (ø)`	⬆️
src/packagedcode/__init__.py	`82.05% <100%> (+3.92%)`	⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b69166...8f94757. Read the comment docs.

* TODO: Think about how to structure data in results Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Make this a nested defaultdict instead of using a tuple for the key Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

* Change summarized_directories codebase attribute to summaries Signed-off-by: Jono Yang <jyang@nexb.com>

* Change summary codebase attribute to be a list instead of an OrderedDict Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Pass files, directories, and special files through is_ignored Signed-off-by: Jono Yang <jyang@nexb.com>

* We now return consolidated components and consolidated packages in two top-level attributes Signed-off-by: Jono Yang <jyang@nexb.com>

* Replace references to codebase.get_root(0) with codebase.root Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update some package parsers with path patterns to ignore * Update tests for get_package_resource Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update about.py with ignorable_path_patterns Signed-off-by: Jono Yang <jyang@nexb.com>

…kit into origin-summary-plugin

Signed-off-by: Jono Yang <jyang@nexb.com>

* Do not combine license-holder components * Add and update tests * Update docs and comments Signed-off-by: Jono Yang <jyang@nexb.com>

* Package parsers can define what resources to ignore when returning package files * Resource and Codebase have been updated to support using this callable * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* If we are in a directory with no majority, we check the child directories if they have any majorities and return those * This prevents us from reported the same majority thing as we come up the codebase * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

* Add new test Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use proper test data as it appears in the real world * Reorder attributes on Consolidation * Update comments Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne

Thank you! See my comment inline. The only significant point is the names and order of the attributes on the Consolidation.

pombredanne · 2019-07-30T20:45:12Z

+
+# Nested ScanCode plugins
+*build*
+*dist*


should this be instead moved to plugins/scancode-consolidate-scan/.gitignore ?
I fear that this may be catching too many things

I've added a .gitignore to scancode-consolidate-scan

pombredanne · 2019-07-30T20:48:47Z

+    core_holders = attr.ib(default=attr.Factory(list))
+    other_holders = attr.ib(default=attr.Factory(list))
+    resources = attr.ib(default=attr.Factory(list))
+


I think I might want instead somthing like this:

consolidated_license_expression = attr.ib(default=None) consolidated_holders = attr.ib(default=attr.Factory(list)) consolidated_copyright = attr.ib(default=None) core_license_expression = attr.ib(default=None) core_holders = attr.ib(default=attr.Factory(list)) other_license_expression = attr.ib(default=None) other_holders = attr.ib(default=attr.Factory(list)) ....

pombredanne · 2019-07-30T20:50:04Z

+
+    def consolidate_core_copyright(self):
+        # TODO: Verify and test that we are generating detectable copyrights
+        holders = list(self.core_holders) + list(self.other_holders)


This is one area where we would want to avoid duplicates (but we still want to have the core before the other)

pombredanne · 2019-07-30T20:50:56Z

+
+    def consolidate_other_copyright(self):
+        # TODO: Verify and test that we are generating detectable copyrights
+        other_holders = list(self.other_holders)


Is wrapping in a list really needed there and elsewhere?

pombredanne · 2019-07-30T20:56:04Z

+
+            discovered_license_expressions = []
+            discovered_holders = []
+            for package_resource in package_resources:


Side note: there is something called key files (from the --classify option) and they should be used when we have a package to craft the things that go in the core vs. other. That could be added later though

pombredanne · 2019-07-30T21:00:30Z

+            if child.extra_data.get('in_package_component'):
+                continue
+            if child.is_file:
+                license_expression = combine_expressions(child.license_expressions)


We need to add a reminder to consider the license match coverage and/or score here in the near future. Otherwise we may end up creating consolidated things that have licenses that were weakly detected.

* Remove plugin specific gitignore from project level gitignore * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

* Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

* Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne · 2019-07-31T20:54:45Z

LGTM. Thank you ++
Merging now.

JonoYang added 11 commits May 28, 2019 18:18

Summarize origin findings to directory level

ca4dcb2

* If a license or copyright is detected in 80% of files in a given directory, that license and copyright is summarized at the directory level Signed-off-by: Jono Yang <jyang@nexb.com>

Summarize detected licenses

691256e

* Register origin summary plugin Signed-off-by: Jono Yang <jyang@nexb.com>

Keep proper count of licenses and copyrights

44c9915

* Define majority to be 50% or more of a directory Signed-off-by: Jono Yang <jyang@nexb.com>

Do not manually keep track of total file count

88bd289

Signed-off-by: Jono Yang <jyang@nexb.com>

Summarize on license expression and holders

d64c886

* Add new flags for summarized rows or if a row is a summary Signed-off-by: Jono Yang <jyang@nexb.com>

Walk codebase again to tag summarized Resources

9291978

Signed-off-by: Jono Yang <jyang@nexb.com>

Refactor plugin

5d2753d

* Group holders and licenses together when counting * Check to see if copyrights or licenses is an attribute instead of requiring the copyright or license option to run the plugin Signed-off-by: Jono Yang <jyang@nexb.com>

Add tests for origin summary plugin

4ec302a

* Increase majority threshold to 0.75 Signed-off-by: Jono Yang <jyang@nexb.com>

Test that null values are not summarized

1be63ad

Signed-off-by: Jono Yang <jyang@nexb.com>

Update help text and comments

cdbf131

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang added 4 commits June 7, 2019 18:47

Group summarized directories

9af6e1e

* TODO: Think about how to structure data in results Signed-off-by: Jono Yang <jyang@nexb.com>

Order summarized directories by license then holder

e9075cb

Signed-off-by: Jono Yang <jyang@nexb.com>

Sort summarized directories

96141a1

Signed-off-by: Jono Yang <jyang@nexb.com>

Unpack summarized_dirs_by_license_and_holder

9d02f36

* Make this a nested defaultdict instead of using a tuple for the key Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the origin-summary-plugin branch from 6053141 to d070252 Compare June 10, 2019 23:37

Create option to set threshold for origin summary

706e3da

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the origin-summary-plugin branch from d070252 to 706e3da Compare June 10, 2019 23:37

JonoYang added 3 commits June 10, 2019 16:53

Update tests

b28e6c0

Signed-off-by: Jono Yang <jyang@nexb.com>

Check summarized directories in tests

e63a14b

Signed-off-by: Jono Yang <jyang@nexb.com>

Create CLI test for custom summarization threshold

8303aee

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the origin-summary-plugin branch from 1c9f6ee to 8303aee Compare June 11, 2019 00:32

Update expected help text

ddc72be

Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne added the summaries label Jun 11, 2019

pombredanne changed the title ~~Origin summary plugin~~ Add origin summary plugin Jun 12, 2019

JonoYang added 4 commits June 12, 2019 18:40

Set origin-summary-threshold to be a float

06a7fd7

* Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Place filesets in summaries codebase attribute

5af6a3f

* Change summarized_directories codebase attribute to summaries Signed-off-by: Jono Yang <jyang@nexb.com>

Create Summary class to store summary info

b1cd77c

* Change summary codebase attribute to be a list instead of an OrderedDict Signed-off-by: Jono Yang <jyang@nexb.com>

Update tests

da1303c

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang added 15 commits July 24, 2019 13:33

Test skip_special argument on is_ignored

f0e728d

* Pass files, directories, and special files through is_ignored Signed-off-by: Jono Yang <jyang@nexb.com>

Refactor Component into two classes

de5589b

* We now return consolidated components and consolidated packages in two top-level attributes Signed-off-by: Jono Yang <jyang@nexb.com>

Set enumerated identifer for ConsolidatedPackages

b17a5e7

* Replace references to codebase.get_root(0) with codebase.root Signed-off-by: Jono Yang <jyang@nexb.com>

Use identifier field instead of name

d4e8eff

Signed-off-by: Jono Yang <jyang@nexb.com>

Add new attribute for ignorable path patterns

9dfcf11

* Update some package parsers with path patterns to ignore * Update tests for get_package_resource Signed-off-by: Jono Yang <jyang@nexb.com>

Add ftfy dependency back to setup.py

b36b173

Signed-off-by: Jono Yang <jyang@nexb.com>

Update expected help text

904952c

* Update about.py with ignorable_path_patterns Signed-off-by: Jono Yang <jyang@nexb.com>

Update expected help text

4ee0872

* Update about.py with ignorable_path_patterns Signed-off-by: Jono Yang <jyang@nexb.com>

Merge branch 'origin-summary-plugin' of github.com:nexB/scancode-tool…

240a337

…kit into origin-summary-plugin

Update expected test files

3ac69ce

Signed-off-by: Jono Yang <jyang@nexb.com>

Update README and comments

77e444e

Signed-off-by: Jono Yang <jyang@nexb.com>

Update tests for consolidation plugin

d013440

* Do not combine license-holder components * Add and update tests * Update docs and comments Signed-off-by: Jono Yang <jyang@nexb.com>

Update package model with ignore_resource

e2b50fc

* Package parsers can define what resources to ignore when returning package files * Resource and Codebase have been updated to support using this callable * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Remove get_package_resources test

a9cf4d2

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the origin-summary-plugin branch from 31074d2 to 75fefb6 Compare July 29, 2019 23:29

JonoYang added 4 commits July 29, 2019 17:10

Return component for different majority children

b678f1a

* Add new test Signed-off-by: Jono Yang <jyang@nexb.com>

Remove unused imports

2a389bb

Signed-off-by: Jono Yang <jyang@nexb.com>

Merge branch 'develop' into origin-summary-plugin

f418d1b

Add consolidate_other_copyright

bb1d846

* Use proper test data as it appears in the real world * Reorder attributes on Consolidation * Update comments Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne approved these changes Jul 30, 2019

View reviewed changes

JonoYang force-pushed the origin-summary-plugin branch from a1493a1 to e6a5f99 Compare July 30, 2019 22:35

Update Consolidation attributes

aea1175

* Remove plugin specific gitignore from project level gitignore * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the origin-summary-plugin branch from e6a5f99 to aea1175 Compare July 30, 2019 22:44

JonoYang added 3 commits July 30, 2019 15:50

Properly generate consolidated copyrights

7c5615c

* Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Simplify consolidated license expression

810b4d2

* Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

Rename resources_count to files_count

8f94757

* Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne merged commit 73e083d into develop Jul 31, 2019

pombredanne deleted the origin-summary-plugin branch July 31, 2019 20:55

Navigation Menu

Uh oh!

Uh oh!

Add origin summary plugin - #1588

Add origin summary plugin#1588
pombredanne merged 154 commits into
developfrom
origin-summary-plugin

JonoYang commented Jun 7, 2019 •

edited

Loading

Uh oh!

codecov Bot commented Jun 7, 2019 •

edited

Loading

Uh oh!

pombredanne left a comment

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

JonoYang Jul 30, 2019

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

Uh oh!

pombredanne Jul 30, 2019

Uh oh!

pombredanne commented Jul 31, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

JonoYang commented Jun 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

JonoYang Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pombredanne Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

pombredanne commented Jul 31, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JonoYang commented Jun 7, 2019 •

edited

Loading

codecov Bot commented Jun 7, 2019 •

edited

Loading