Inspect_manifest pipeline error for get_manifest_inputs()

[mjherzog]

Describe the bug
I uploaded a tar.gz archive of SCWB 4.0 for an SCIO project with the inspect_manifest pipeline. The archive contains 2 manifest files: package.json and package-lock.json.

The pipeline failed with the error:

No package type found for /var/scancodeio/workspace/projects/scwb-packages-test-02-aa820b26/input/scancode-workbench-4.0.0.tar.gz

Traceback:
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 120, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/inspect_manifest.py", line 67, in get_packages_from_manifest
    packages = resolve.resolve_packages(input_location)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/resolve.py", line 49, in resolve_packages
    raise Exception(f"No package type found for {input_location}")

I also created a project where I uploaded just the package-lock.json file and that worked.

System configuration

• Which version of ScanCode.io are you running? v3.5.2
• Are you running the app using Docker? Yes
• On which OS? Ubuntu

Expected behavior
I expected the inspect_manifest pipeline to find the package-lock.json file in the archive and report the npm packages (1448).

[tdruez]

@mjherzog you can upload both unarchived files as a workaround.

[mjherzog]

I already did that and it worked, but we cannot expect a team to manually find and upload the manifest files in a codebase. This needs to be automated and the current doc indicates that the pipeline should handle the extraction.

[pombredanne]

I think what we need mostly is a way to inspect a codebase for packages only.

1. inspect_manifest should inspect an archive, BUT
2. the archive may contain more than manifests, so we need really something different

Here is the plan I suggest:

• rename inspect_manifest to inspect_packages Rename multiple pipelines #1044 #1053 with a description that this inspects application package manifests and lockfiles and creates package and dependencies accordingly. Get it to do the equivalent of a scancode --package or the upcoming scancode --purl
• create a new pipeline resolve_dependencies that will resolve deps with inspectors (only on pypi for now) and only do this, nothing else (this will not have any steps otherwise)
• drop scan_codebase_packages that has a confusing name and would now be the same as inspect_packages