RFC: Design improved relationships for Package, Dependencies, Embeds and Requirements

[pombredanne]

We need to design improved models for Package, Dependencies, Embeds and Requirements.

Some background:

• What we routinely calls "dependencies" are not really dependencies: more often these are requirements that are "resolved" to actual concrete dependencies. To really "depend" on a software package, you need to know the exact resolved version. On the other hand, a requirement expresses the need for some range of versions (even if it's a range of a single version).

• We should distinguish a "require" relationship between packages vs. an actual "resolved" dependency to a concrete version, and track if the resolved dependency is present in the codebase or not.

• Dependencies (or requirements) may be only documented in a package manifest (like pom.xml) or a lockfile (like package-lock.json), or also be present in the codebase.

• We need to track also when a package contains another package (or for that matter when an app contains or embeds packages).

Therefore we may have three levels of tree-like data we need to track for packages:

• "potential" requirements (we call these dependencies today). This is a logically a flat list of PURLs with VERS attached to a package instance.
• "resolved" dependencies (we call these dependencies today too, but sometimes also packages). This is logically a tree (or graph) of PURLs with a resolved version. Each such dependency is derived and satisfies one or more "requirement".
• "embeds" are packages embedded in other packages, e.g., they are physically copied, vendor and or built into a product codebase. These could have been the results of provisioning/fetching resolved dependencies, or could be a straight copy. These are not much tracked as such today.

We have agreed on some key model updates, based on discussions here and in our weekly calls:

• Enhance ScanCode.io models to support relating dependencies to packages
• Enhance ScanCode Toolkit models to support relating dependencies to packages
• DJC: Enhance DejaCode models to support relating dependencies to packages dejacode#138
• Enhance PurlDB models to support relating dependencies to packages purldb#485
• Enhance python-inspector models to support relating dependencies to packages using the new models
• Enhance nuget-inspector models to support relating dependencies to packages using the new models
• Create dependency-inspector to relate dependencies to packages using the new models

The https://github.com/nexB/dependency-inspector tickets to create lockfiles are:

• Create a yarn.lock lock file with yarn dependency-inspector#3
• Create lock file with pnpm dependency-inspector#5
• Create a package-json.lock lock file with npm dependency-inspector#4
• Create SwiftPM lockfile dependency-inspector#12
• Create cocoapods lockfile dependency-inspector#10
• Create Python frozen requirements file dependency-inspector#8
• Create .Net lock file with dotnet restore and other built in commands dependency-inspector#11

Then for the effective resolution of dependencies as a tree from manifests and lockfiles:

• npm: SCIO: Lock and resolve locked npm dependencies from yarn and npm #1201 and SCIO: Resolve npm dependencies based on lockfiles #1237
• pypi: SCIO: Detect frozen Python requirements locked with deplock #1262
• nuget: SCIO: Detect frozen .NuGet dependencies locked with deplock #1263
• cocoapods: SCIO: Detect and resolve locked Cocoapods requirements #1279
• swift: SCIO: Detect and resolve locked Swift requirements #1278

And beyond:

• Extend Lockfile checker and generator dependency-inspector#13

Here are also some related issues, but not directly need to complete this feature:

• Support packages embedded in other packages
  • RFC: Introduce "primary package" vs. "embedded- or sub-packages" scancode-toolkit#2418
  • RFC: Introduce idea of embedded package purldb#177
  • Store if source code file paths contain "third party" hints purldb#163
• Improve the definition of a package and dependency
  • Definitions for DiscoveredPackage and DiscoverdDependency are very different #885
  • Should a dependency tree be scoped to a single project ? #888
• Introduce successor packages Introduce the concept of a "successor package" to the purldb data model purldb#175
• and also https://github.com/nexB/python-inspector, https://github.com/nexB/nuget-inspector and other inspectors.

[pombredanne]

Note that this is as much for ScanCode.io as it is for ScanCode Toolkit, DejaCode and PurlDB

[AyanSinhaMahapatra]

Summarizing the discussion we had on the community call on this issue:

We need to have dependencies also as packages, and not
as a separate model (with the skinny purl-only package data and optionally
resolved packages nested inside) we have today. And dependencies here
would be preserved as relationships between these packages, and the actual
primary package -> secondary packages (either dependencies/embedded packages)
would also be preserved by creating relationships between them.

Here we have some types of packages:

1. Primary package instances
2. Resolved package dependency with package data/files
3. Resolved package dependency without package data/files (just PURL)
4. Secondary/Embedded package (files embedded in primary package files)
5. Package requirements (package + version range)

This approach has some benefits:

• We don't have to change massively our package models, just some attribute addition/deletion
  as we are changing how we report the data within those models, and the output format
  remains the same essentially.
• Resolved packages won't be nested, everything will be there as package instances.
• This is straightforward, as dependency is really a relationship and not a separate model.
• Similar to what users expect/other tools provide, and essentially what we will have if
  we scan a package with pre-resolved dependencies present in the code tree, there too
  since we have the packages detected as instances, we have them as packages.

For example, in SCIO we could have filters for primary packages, secondary/embedded packages,
so that these are differentiated and looked at separately as required, and the dependency tab
could have the relations between them.

Two possible implementations in this:

a) We have everything (packages, resolved/embedded dependencies, dependency purls and even requirements)
returned as Packages.

• Would be less confusing and straightforward, having everything stored similarly.
• We can do a resolve requirements to the oldest version available within
  the required versions, or something similar, to have a proper package instance.

b) We have everything else but requirements (packages, resolved/embedded dependencies, dependency purls)
as packages. Requirements (does not have exact version, therefore does not point to a package)

• Requirements do not have a version, so these aren't really package instances as
  they are not concrete packages which can be identified in a unique way. So these
  should not be confused with packages.
• Even for dependency purls we can get package metadata/download the packages and scan them, but we
  cannot do the same for dependency requirements, as they are not package instances.
• If we try to resolve these dependency requirements incorrectly (to the oldest version available within
  the required versions) this would be incorrect and misleading for any compliance. As package licenses
  could change between versions and without correct versions we cannot check vulnerabilities. So these
  are essentially false positives.

cc @pombredanne @tdruez

[pombredanne]

The latest design is to go in this direction:

• We are transforming the DiscoveredDependency model to add a foreign key to the corresponding resolved DiscoveredPackage something like a "resolved_to" relationship
• We will have a dependency resolution process in a pipeline where we look at the dependencies and resolve them to actual found packages or can also use a resolver, even if very crude.
• We may also track if a package is physically found on disk in a codebase
• This can stay optional
• We will need to ensure that we are doing the right thing too on the scancode toolkit side
• The same solution will be then propagated to PurlDB and possibly to DejaCode

See also attached:

To clarify, we would go with option 1:

1. We do not create a "resolved to" until we "run" a resolution

2. We resolve as below possibly in a dumb way. We are aware that looking for the latest version of a package is a somewhat dumb "mock resolution" and may not have a lot of value

3. When we have a lockfile (that contains pinned dependencies) we could create the "resolved to" Package at the same time

4. When we have already existing Packages and a Dependencies, we should be able to relate as a resolution the existing Packages (a sort of mock resolution

[pombredanne]

@AyanSinhaMahapatra I had some other notes too... did you capture some?

[pombredanne]

We have a mostly working PR with #896 by @Hritik14