Add support for package instances

[JonoYang]

In scancode-toolkit, we report each specific instance with a package_uid to differentiate multiple detection of the same package in a codebase. We should implement this idea in scancode.io when we discover packages in pipelines.

[tdruez]

@JonoYang is the recent merged changes address this? Or is there more we can do?

[JonoYang]

@tdruez The recent changes adds the package_uid field to DiscoveredPackage model, which is populated if we use the scan_package pipeline. However, we use the function scan_for_application_packages (https://github.com/nexB/scancode.io/blob/2d342fa1a1b4f06fea9be73298a00e627cbb6a46/scanpipe/pipes/scancode.py#L311) during the scan_for_application_packages step on the docker, rootfs, and scan_codebase pipelines. This function doesn't assign a package_uid for discovered application packages, which we would need to add to bring scancode.io application package scanning to be more in line with scancode-toolkit.

The package scanning function of scancode is now two steps, where we first scan the individual Resources for package data, and then afterwards we assemble the package data from multiple resources and assign package instances. We would need a way to do the package assembly step in scancode.io.

[JonoYang]

After talking to @pombredanne, it may be necessary to create a new model that for Package and Dependency instances. The current charts in the project detail view show the total number of Packages or Dependencies discovered in a Project. With the introduction of Package and Dependency instances, the Project chart stats are thrown off because we could have multiple copies of the same packages and dependencies.

1. Using an instance for packages and deps as an intermediate model
   Pkg -> PkgInstance -> Res
   Pkg -> PkgInstance -> Dep -> DepInstance - >(and when we resolve -> Pkg or PkgInstance)

Here, we can show each unique Package and Dependency, then show the individual instances of the packages, and then show which resources correspond to which Package instance.

[pombredanne]

Right now the latest thinking would be to treat the current DiscoveredPackage record as being a package instance. We would add a UID and we would display each copy of a package as it's own. Duplication in the UI and reports is something that is not exceptional but not the main case either and we can solve this in the UI in future.

[pombredanne]

Should datafile_paths be an actual relationship to a CodebaseResource? ... IMHO yes and we are missing it https://github.com/nexB/scancode-toolkit/blob/bfa6d9632c9ef63250267c8ce3be2d5ddae1a9fc/src/packagedcode/models.py#L1224