Add system packages support in the new packages model

docs/source/contribute/contrib_dev.rst

@@ -109,7 +109,30 @@ These are enabled by adding a ``--test-suite`` option to the pytest command.
   extensive data-driven and data validations (for package, copyright and license
   detection)
 
+In some cases we need to regenerate test data when expected behavious/result data
+structures change, and we have an environement variable to regenerate test data.
+`SCANCODE_REGEN_TEST_FIXTURES` is present in `scancode_config` and this can be
+set to regenerate test data for specific tests like this:
 
+``SCANCODE_REGEN_TEST_FIXTURES=yes pytest -vvs tests/packagedcode/test_package_models.py``
+
+This command will only regenerate test data for only the tests in `test_package_models.py`,
+and we can further specify the tests to regen by using more pytest options like `--lf` and
+`-k test_instances`.
+
+If test data is regenerated, it is important to review the diff for test files and
+carefully go through all of it to make sure there are no unintended changes there,
+and then commit all the regenerated test data.
+
+To help debug in scancode, we use logging. There are different environement variables
+you need to set to turn on logging. In packagedcode::
+
+``SCANCODE_DEBUG_PACKAGE=yes pytest -vvs tests/packagedcode/ --lf``
+
+Or set the ``TRACE`` variable to ``True``. This enables ``logger_debug`` functions
+logging variables and shows code execution paths by logging and printing the logs
+in the terminal. If debugging full scans run by click, you have to raise exceptions
+in addition to setting the TRACE to enable logging.
 
 .. _scancode_toolkit_development_thirdparty_libraries:
 

setup-mini.cfg

@@ -69,7 +69,8 @@ install_requires =
     chardet >= 3.0.0
     click >= 6.7, !=7.0
     colorama >= 0.3.9
-    commoncode >= 30.0.0
+    commoncode >= 30.1.1
+    container-inspector >= 30.0.0
     debian-inspector >= 30.0.0
     dparse2 >= 0.6.0
     fasteners
@@ -175,14 +176,13 @@ scancode_scan =
 # module for details and doc.
 scancode_post_scan =
     summary = summarycode.summarizer:ScanSummary
-    summary2 = summarycode.summarizer2:ScanSummary
-    summary-keeping-details = summarycode.summarizer:ScanSummaryWithDetails
-    summary-key-files = summarycode.summarizer:ScanKeyFilesSummary
-    summary-by-facet = summarycode.summarizer:ScanByFacetSummary
+    tallies = summarycode.tallies:Tallies
+    tallies-with-details = summarycode.tallies:TalliesWithDetails
+    tallies-key-files = summarycode.tallies:KeyFilesTallies
+    tallies-by-facet = summarycode.tallies:FacetTallies
     license-clarity-score = summarycode.score:LicenseClarityScore
     license-policy = licensedcode.plugin_license_policy:LicensePolicy
     mark-source = scancode.plugin_mark_source:MarkSource
-    classify-package = summarycode.classify:PackageTopAndKeyFilesTagger
     is-license-text = licensedcode.plugin_license_text:IsLicenseText
     filter-clues = cluecode.plugin_filter_clues:RedundantCluesFilter
     consolidate = summarycode.plugin_consolidate:Consolidator

setup.cfg

@@ -70,6 +70,7 @@ install_requires =
     click >= 6.7, !=7.0
     colorama >= 0.3.9
     commoncode >= 30.1.1
+    container-inspector >= 30.0.0
     debian-inspector >= 30.0.0
     dparse2 >= 0.6.0
     fasteners
@@ -184,7 +185,6 @@ scancode_post_scan =
     license-clarity-score = summarycode.score:LicenseClarityScore
     license-policy = licensedcode.plugin_license_policy:LicensePolicy
     mark-source = scancode.plugin_mark_source:MarkSource
-    classify-package = summarycode.classify:PackageTopAndKeyFilesTagger
     is-license-text = licensedcode.plugin_license_text:IsLicenseText
     filter-clues = cluecode.plugin_filter_clues:RedundantCluesFilter
     consolidate = summarycode.plugin_consolidate:Consolidator

src/formattedcode/output_csv.py

@@ -6,6 +6,7 @@
 # See https://github.com/nexB/scancode-toolkit for support or download.
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
+import attr
 import csv
 
 import saneyaml
@@ -231,6 +232,8 @@ def get_package_columns(_columns=set()):
 
     from packagedcode.models import PackageData
 
+    package_data_fields = [field.name for field in attr.fields(PackageData)]
+
     # exclude some columns for now that contain list of items
     excluded_columns = {
         # list of strings
@@ -252,7 +255,7 @@ def get_package_columns(_columns=set()):
         'notice_url',
     ]
 
-    fields = PackageData.fields() + extra_columns
+    fields = package_data_fields + extra_columns
     _columns = set(f for f in fields if f not in excluded_columns)
     return _columns
 

src/packagedcode/README.rst

@@ -31,19 +31,24 @@ Taking Python as a main example a package can exist in multiple forms:
     file type with metadata such as Windows DLLs. Additional markers may also include
     "namespaces" such as Java or Python imports, C/C++ namespace declarations.
 
-2. **parse and collect the package manifest(s)** metadata. For Python, this means
+2. **parse and collect the package datafile or manifest(s)** metadata. For Python, this means
 extracting name, version, authorship, declared licensing and declared dependencies as
 found in the any of the package descriptor files (e.g. a `setup.py` file,
 `requirements` file(s) or any of the `*-dist-info` or `*-egg-info` dir files such as
-a `metadata.json`). Other package formats have their own metatada that may be more or
+a `metadata.json`). Other package datafile formats have their own metatada that may be more or
 less comprehensive in the breadth and depth of information they offer (e.g.
 `.nuspec`, `package.json`, `bower.json`, Godeps, etc...). These metadata include the
 declared dependencies (and in some cases the fully resolved dependencies too such as
 with Gemfile.lock). Finally, all the different packages formats and data are
 normalized and stored in a common data structure abstracting the small differences of
 naming and semantics that may exists between all the different package formats.
 
-Once collected, these data are then injected in the `packages` section of the scan. 
+Once collected, these data are then injected in the `package_data` section of a file scan
+for each recognized package datafile. 
+
+3. **assemble multiple package datafile** as top level packages.
+
+
 
 What code in `packagedcode` is not meant to do: