On-demand JAR processing

[JonoYang]

We request through an API the sha1 and purl fields of the Package we are looking up

• If we have a checksum, we look in the PurlDB for it. What do we do with the purl fields? If we request the checksum and we get npm thing back
• If we have a match, return it
  • If there is no scans for these matches, trigger scan requests for all the package versions of this package (source and binary, vcs): we are only triggering this for the requested version.
• If no match, we do a live query on upstream repo using checksum and purl type: this is done by get_or_fetch_package API endpoint
  • If there is a match on the upstream repo, we get package metadata (pom + chain of parent poms)
• If we have no checksum, we're going to use the bits we have from the purl fields to do a query based on the purl. (That's done in get_package and get_or_fetch_package API endpoints)
  • If we have a type, namespace, and name, we can collect all the versions and focus on returning the latest one. We do this for for Maven alright now.
  • If we have a version, then we can look it up more specifically
• When we get the info, we store the package and return it to the user. (get_or_fetch_package API endpoint)
  • Trigger scan requests for packages that we create (done by get_package or get_or_fetch_package API endpoint)

These are separate and tracked in #88

• Trigger mining, matching, scanning of the dependency trees, possibly with priority for runtime dependencies over others
• Trigger mining, matching, scanning for newer versions of packages that were matched
• Trigger mining, matching, scanning for older versions of packages that were matched
• Create a watcher that looks periodically for new versions of a package that was requested, may be by registering interest in this package

[JonoYang]

if we have a jar, consider looking up the sha1 on maven if we don't have the data on packagedb

[JonoYang]

There are a few ways to populate the PackageDB using the on-demand queue. One is the load_priority_queue management command, which takes in a scancode-toolkit/scancode.io JSON scan of JARs, looks up the purl for these sha1s, and then creates new entries on the on-demand queue for these packages.

The other way would be to use the get_package or get_or_fetch_package endpoints, where a purl is passed in as a query parameter and if that package exists in the PackageDB, it is returned to the user. If not, the package is added to the on demand queue (in the case of get_package) or we go and directly fetch the metadata during the request (in the case of get_or_fetch_package)

In both cases, if we create a new package, that package is also added to the scan queue for scanning.

[pombredanne]

This is complete and the follow ups are in #87 and #88
Thank you.