Run a package-only scan on a codebase as a new pipeline - Proactively scan and review all my packages

[pombredanne]

Following #720 and aboutcode-org/purldb#87 we need to have a pipeline that would only populate the packages and dependencies (and eventually later on also resolve dependencies)

The goal is to ensure that the purlDB is kept always up-to-date with the set of packages effectively used in a development codebase.

The overall process would be:

• On commit or on a daily schedule, create a new project and run pipeline #828
  • There, either clone and checkout or download the latest commit of a codebase
• Run a new pipeline that only collects the packages minimally, for now using whatever pipes we have.
• In the future we should be using a skinny PURL only approach
  • Optionally only collect the PURL of packages, source packages and dependencies in ScanCode Toolkit scancode-toolkit#3464
• Once this pipeline is complete, run a second "populate_purldb" pipeline https://github.com/nexB/scancode.io/blob/3ed1e9e0fd7bf0fd9e978522e6365e33ee156ef8/scanpipe/pipelines/populate_purldb.py .... this will trigger indexing and scanning there.

Separately I would like to have a way to determine if any of the package populated in the purlDB here has any license or origin issues based on data clarity and accuracy (using summaries, scores, --todo, package set, policies, compliance alerts, etc. and TBD )and I want to be alerted to review and eventually curate the issues that were uncovered, by exception.

Ideally there would be some minimal request/ticket system where a form would be posted for any item that would need further review. The ideal outcome would be to push and store a curated version of the package data (possibly in the purldB as part of a package set with a "curated" type) , or some ABOUT file that I could download to save in my codebase.