Merge latest develop

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne

.gitignore

@@ -11,6 +11,7 @@
 /src/*.egg-info
 *.egg-info
 !tests/packagedcode/data/pypi/source-package/pip-22.0.4/src/pip.egg-info
+!tests/packagedcode/data/pypi/unpacked_sdist/prefer-egg-info-pkg-info/celery/celery.egg-info
 /dist
 /build
 /bin

CHANGELOG.rst

@@ -14,6 +14,30 @@ Package detection:
   - OpenWRT packages.
   - Yocto/BitBake .bb recipes.
 
+- Update ``GemfileLockParser`` to track the gem which the Gemfile.lock is for,
+  which we assign to the new ``GemfileLockParser.primary_gem`` field. Update
+  ``GemfileLockHandler.parse()`` to handle the case where there is a primary gem
+  detected from a gemfile.lock. If there is a primary gem, a single ``Package``
+  is created and the detected gem data within the gemfile.lock are assigned as
+  dependencies. If there is no primary gem, then all of the dependencies are
+  collected into Package with no name and yielded.
+
+  https://github.com/nexB/scancode-toolkit/issues/3072
+
+- Fix issue where dependencies were not reported when scanning an extracted
+  Python project by modifying ``BaseExtractedPythonLayout.assemble()`` to favor
+  using package data from a PKG-INFO file from an egg-info directory. Package
+  data from a PKG-INFO file from an egg-info directory contains the dependency
+  information collected from the requirements.txt file along side PKG-INFO.
+
+  https://github.com/nexB/scancode-toolkit/issues/3083
+
+- Fix issue where we were returning incorrect purl package ``type`` for cocoapods.
+  ``pods`` was being returned  as a purl type for cocoapods, it should be
+  ``cocoapods`` instead.
+  https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#cocoapods
+
+  https://github.com/nexB/scancode-toolkit/issues/3081
 
 License detection:
 ~~~~~~~~~~~~~~~~~~~

docs/source/explanations/overview.rst

@@ -27,18 +27,18 @@ ScanCode-Toolkit performs the scan on a codebase in the following steps :
 
 Scan results are provided in various formats:
 
-- a JSON file simple or pretty-printed,
-- SPDX tag value or XML, RDF formats,
-- CSV,
-- a simple unformatted HTML file that can be opened in browser or as a spreadsheet.
+* a JSON file simple or pretty-printed,
+* SPDX tag value or XML, RDF formats,
+* CSV,
+* a simple unformatted HTML file that can be opened in browser or as a spreadsheet.
 
 For each scanned file, the result contains:
 
-- its location in the codebase,
-- the detected licenses and copyright statements,
-- the start and end line numbers identifying where the license or copyright was found in the
+* its location in the codebase,
+* the detected licenses and copyright statements,
+* the start and end line numbers identifying where the license or copyright was found in the
   scanned file, and
-- reference information for the detected license.
+* reference information for the detected license.
 
 For archive extraction, ScanCode uses a combination of Python modules, 7zip and libarchive/bsdtar
 to detect archive types and extract these recursively.

src/licensedcode/data/licenses/ccrc-1.0.LICENSE

@@ -0,0 +1,47 @@
+Common Cure Rights Commitment
+Version 1.0
+ 
+Before filing or continuing to prosecute any legal proceeding or claim
+(other than a Defensive Action) arising from termination of a Covered
+License, we commit to extend to the person or entity ('you') accused
+of violating the Covered License the following provisions regarding
+cure and reinstatement, taken from GPL version 3. As used here, the
+term 'this License' refers to the specific Covered License being
+enforced.
+
+    However, if you cease all violation of this License, then your
+    license from a particular copyright holder is reinstated (a)
+    provisionally, unless and until the copyright holder explicitly
+    and finally terminates your license, and (b) permanently, if the
+    copyright holder fails to notify you of the violation by some
+    reasonable means prior to 60 days after the cessation.
+
+    Moreover, your license from a particular copyright holder is
+    reinstated permanently if the copyright holder notifies you of the
+    violation by some reasonable means, this is the first time you
+    have received notice of violation of this License (for any work)
+    from that copyright holder, and you cure the violation prior to 30
+    days after your receipt of the notice.
+
+We intend this Commitment to be irrevocable, and binding and
+enforceable against us and assignees of or successors to our
+copyrights.
+
+Definitions
+
+'Covered License' means the GNU General Public License, version 2
+(GPLv2), the GNU Lesser General Public License, version 2.1
+(LGPLv2.1), or the GNU Library General Public License, version 2
+(LGPLv2), all as published by the Free Software Foundation.
+
+'Defensive Action' means a legal proceeding or claim that We bring
+against you in response to a prior proceeding or claim initiated by
+you or your affiliate.
+
+'We' means each contributor to this repository as of the date of
+inclusion of this file, including subsidiaries of a corporate
+contributor.
+
+This work is available under a Creative Commons Attribution-ShareAlike
+4.0 International license (https://creativecommons.org/licenses/by-sa/4.0/).
+

src/licensedcode/data/licenses/gpl-2.0-classpath.LICENSE

@@ -0,0 +1,9 @@
+This library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with this library; see the file COPYING.  If not, write to the Free Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+Linking this library statically or dynamically with other modules is making a combined work based on this library. Thus, the terms and conditions of the GNU General Public License cover the whole combination.
+
+As a special exception, the copyright holders of this library give you permission to link this library with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you also meet, for each linked independent module, the terms and conditions of the license of that module. An independent module is a module which is not derived from or based on this library. If you modify this library, you may extend this exception to your version of the library, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version.

src/licensedcode/data/licenses/gpl-2.0-independent-module-linking.LICENSE

@@ -0,0 +1,40 @@
+This library is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This library is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License along
+with this library; see the file COPYING.  If not, write to the Free
+Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA
+02110-1301, USA.
+
+The library is released under the GPL with the following exception:
+
+Linking this library statically or dynamically with other modules is
+making a combined work based on this library. Thus, the terms and
+conditions of the GNU General Public License cover the whole
+combination.
+
+As a special exception, the copyright holders of this library give you
+permission to link this library with independent modules to produce an
+executable, regardless of the license terms of these independent
+modules, and to copy and distribute the resulting executable under terms
+of your choice, provided that you also meet, for each linked independent
+module, the terms and conditions of the license of that module. An
+independent module is a module which is not derived from or based on
+this library. If you modify this library, you may extend this exception
+to your version of the library, but you are not obligated to do so. If
+you do not wish to do so, delete this exception statement from your
+version.
+
+Note The exception is changed to reflect the latest GNU Classpath
+exception. Older versions of #ziplib did have another exception, but the
+new one is clearer and it doesn't break compatibility with the old one.
+
+Bottom line In plain English this means you can use this library in
+commercial closed-source applications.

src/licensedcode/data/licenses/gpl-2.0-library.LICENSE

@@ -0,0 +1,22 @@
+The software in this package is distributed under the GNU General Public
+License with the "Library Exception" described below. A copy of GNU
+General Public License (GPL) is included in this distribution, in the
+file LICENSE-GPL.txt. All the files distributed under GPL also include
+the following special exception:
+
+As a special exception, the copyright holders of this library give you
+permission to link this library with independent modules to produce an
+executable, regardless of the license terms of these independent
+modules, and to copy and distribute the resulting executable under terms
+of your choice, provided that you also meet, for each linked independent
+module, the terms and conditions of the license of that module. An
+independent module is a module which is not derived from or based on
+this library. If you modify this library, you may extend this exception
+to your version of the library, but you are not obligated to do so. If
+you do not wish to do so, delete this exception statement from your
+version.
+
+As such, this software can be used to run free as well as proprietary
+applications and applets. Modifications made to the classes in this
+distribution must however be distributed under the GPL, optionally with
+the same exception as above.

src/licensedcode/data/licenses/gpl-2.0-openjdk.LICENSE

@@ -0,0 +1,38 @@
+This library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with this library; see the file COPYING.  If not, write to the Free Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"CLASSPATH" EXCEPTION TO THE GPL VERSION 2
+
+Certain source files distributed by Sun Microsystems, Inc. are subject to the following clarification and special exception to the GPL Version 2, but only where Sun has expressly included in the particular source file's header the words
+
+"Sun designates this particular file as subject to the "Classpath" exception as provided by Sun in the License file that accompanied this code."
+
+Linking this library statically or dynamically with other modules is making a combined work based on this library. Thus, the terms and conditions of the GNU General Public License Version 2 cover the whole combination.
+
+As a special exception, the copyright holders of this library give you permission to link this library with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you also meet, for each linked independent module, the terms and conditions of the license of that module. An independent module is a module which is not derived from or based on this library. If you modify this library, you may extend this exception to your version of the library,  but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version. 
+
+OPENJDK ASSEMBLY EXCEPTION
+
+The OpenJDK source code made available openjdk.dev.java.net ("OpenJDK Code") is distributed under the terms of the GNU General Public License <http://www.gnu.org/copyleft/gpl.html> version 2 only ("GPL2"), with the following clarification and special exception.
+
+Linking this OpenJDK Code statically or dynamically with other code is making a combined work based on this library. Thus, the terms and conditions of GPL2 cover the whole combination.
+
+As a special exception, Sun gives you permission to link this OpenJDK Code with certain code licensed by Sun as indicated at http://openjdk.java.net/legal/exception-modules-2007-05-08.html ("Designated Exception Modules") to produce an executable, regardless of the license terms of the Designated Exception Modules, and to copy and distribute the resulting executable under GPL2, provided that the Designated Exception Modules continue to be governed by the licenses under which they were offered by Sun.
+
+As such, it allows licensees and sublicensees of Sun's GPL2 OpenJDK Code to build an executable that includes those portions of necessary code that Sun could not provide under GPL2 (or that Sun has provided under GPL2 with the Classpath exception). If you modify or add to the OpenJDK code, that new GPL2 code may still be combined with Designated Exception Modules if the new code is made subject to this exception by its copyright holder.
+
+from http://openjdk.java.net/legal/exception-modules-2007-05-08.html 
+OpenJDK Designated Exception Modules
+8 May 2007
+For purposes of those files in the OpenJDK distribution that are subject to the Assembly Exception, the following shall be deemed Designated Exception Modules:
+
+Those files in the OpenJDK distribution available at openjdk.java.net, openjdk.dev.java.net, and download.java.net to which Sun has applied the Classpath Exception,
+
+Any of your derivative works of #1 above, to the extent you license them under the GPLv2 with the Classpath Exception as defined in the OpenJDK distribution available at openjdk.java.net, openjdk.dev.java.net, or download.java.net,
+
+Any files in the OpenJDK distribution that are made available at openjdk.java.net, openjdk.dev.java.net, or download.java.net under a binary code license, and
+
+Any files in the OpenJDK distribution that are made available at openjdk.java.net, openjdk.dev.java.net, or download.java.net under an open source license other than GPL, and your derivatives thereof that are in compliance with the applicable open source license.

src/licensedcode/data/licenses/gpl-2.0-oracle-openjdk.LICENSE

@@ -0,0 +1,28 @@
+This library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with this library; see the file COPYING.  If not, write to the Free Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"CLASSPATH" EXCEPTION TO THE GPL
+
+Certain source files distributed by Oracle America and/or its affiliates are
+subject to the following clarification and special exception to the GPL, but
+only where Oracle has expressly included in the particular source file's header
+the words "Oracle designates this particular file as subject to the "Classpath"
+exception as provided by Oracle in the LICENSE file that accompanied this code."
+
+    Linking this library statically or dynamically with other modules is making
+    a combined work based on this library.  Thus, the terms and conditions of
+    the GNU General Public License cover the whole combination.
+
+    As a special exception, the copyright holders of this library give you
+    permission to link this library with independent modules to produce an
+    executable, regardless of the license terms of these independent modules,
+    and to copy and distribute the resulting executable under terms of your
+    choice, provided that you also meet, for each linked independent module,
+    the terms and conditions of the license of that module.  An independent
+    module is a module which is not derived from or based on this library.  If
+    you modify this library, you may extend this exception to your version of
+    the library, but you are not obligated to do so.  If you do not wish to do
+    so, delete this exception statement from your version.

src/licensedcode/data/licenses/ms-refl.LICENSE

@@ -0,0 +1,25 @@
+This license governs use of the accompanying software. If you use the software, you accept this license. If you do not accept the license, do not use the software.
+
+1. Definitions
+The terms "reproduce," "reproduction" and "distribution" have the same meaning here as under U.S. copyright law.
+
+"You" means the licensee of the software.
+
+"Your company" means the company you worked for when you downloaded the software.
+
+"Reference use" means use of the software within your company as a reference, in read only form, for the sole purposes of debugging your products, maintaining your products, or enhancing the interoperability of your products with the software, and specifically excludes the right to distribute the software outside of your company.
+
+"Licensed patents" means any Licensor patent claims which read directly on the software as distributed by the Licensor under this license.
+
+2. Grant of Rights
+(A) Copyright Grant- Subject to the terms of this license, the Licensor grants you a non-transferable, non-exclusive, worldwide, royalty-free copyright license to reproduce the software for reference use.
+
+(B) Patent Grant- Subject to the terms of this license, the Licensor grants you a non-transferable, non-exclusive, worldwide, royalty-free patent license under licensed patents for reference use.
+
+3. Limitations
+(A) No Trademark License- This license does not grant you any rights to use the Licensor's name,
+    logo, or trademarks.
+
+(B) If you begin patent litigation against the Licensor over patents that you think may apply to the software (including a cross-claim or counterclaim in a lawsuit), your license to the software ends automatically.
+
+(C) The software is licensed "as-is." You bear the risk of using it. The Licensor gives no express warranties, guarantees or conditions. You may have additional consumer rights under your local laws which this license cannot change. To the extent permitted under your local laws, the Licensor excludes the implied warranties of merchantability, fitness for a particular purpose and non-infringement.