Parse additional dependency/lock files

[JonoYang]

We should parse Gemfile.lock and go.mod files for package info

[JonoYang]

We have a gemfile.lock parser, but it is not used https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/gemfile_lock.py

[ritiek]

Perhaps Cargo.lock too!

[pombredanne]

@ritiek sure thing... welcome back! do you want to tackle this? (Note also that GSoC project idea https://github.com/nexB/aboutcode/wiki/Project-Ideas-DependentCode-A-Mostly-Universal-Package-Dependency-Resolver )

[ritiek]

@pombredanne Hi, and thanks! Sure, I'd love to take this on!

I took a look at the GSoC idea page you linked. PURL seems a very nice idea. In the past when I worked with PlasmaPy, it quickly became clear how much of a pain it can be working on parsing large particle datasets with no well-defined conventions. And then we discovered about openPMD, which is on a mission to create a universal convention for labeling in datasets. PURL and openPMD are very different in terms of what they do but it seems like they share the same philosophy. 👍

What I understand from:

The goal of this project is to create a (mostly) universal package dependency resolution tool that should leverage the detected packages and package dependencies from ScanCode as Package URLs

Is that we'd like DependentCode, when fed ScanCode's scan result, to list all of package's dependencies in their respective PURL format?

DependentCode would provide dependency resolution to get all other transitive dependencies

By transitive dependencies do we mean such as for example, ScanCode depends on requests and requests in turn depends on urllib3, so does that make urllib3 a transitive dependency to ScanCode?

I'm also not sure what "dependency resolution" means here?

Such as for example, let's say we have a package A. This package A has two direct dependencies - package B, and package C with the constraint C>=1.1.0. But package B also has a direct dependency to the same package C with the constraints C>=1.0.0 && C<1.2.0. Does dependency resolution mean here to use only one version of package C that satisfies the constraints for both package A and package B (which would be C>=1.1.0 && C<1.2.0 )?

I also skimmed through README of https://github.com/heremaps/oss-review-toolkit, but it seems like some of our projects have a bit of overlap with this project? Such as scanning for licenses.

Overall, DependentCode (and PURL) sure seems interesting. I'd love to submit a proposal to work on it this summer!

[MankaranSingh]

Sorry for inconvenience, the above references were made by mistake due to wrong rebasing while signing off my commits for another pull request.