Resolve dependencies and improve JS support by AyanSinhaMahapatra · Pull Request #3779 · aboutcode-org/scancode-toolkit

AyanSinhaMahapatra · 2024-05-16T14:52:58Z

This PR adds support for resolving package dependencies with an initial focus on JavaScript npm packages as support by npm and yarn.

In particular:

It can resolve dependencies per SCTK: resolve npm dependencies based on lockfiles, using a new option #3780 as designed in RFC: Design improved relationships for Package, Dependencies, Embeds and Requirements scancode.io#1066
It adds new support for pnpm. Support pnpm #3766
It adds support for npm workspaces (and their impact in yarn, npm and pnpm). SCTK: Add support for npm workspaces #3746
It track private packages. Track when a package or dependency is private #3102

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Updated documentation pages (if applicable)
Updated CHANGELOG.rst (if applicable)

Add parsers for pnpm-lock.yaml v5 and v6, and shrinkwrap.yaml specs with examples and package assembly. Reference: #3766 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Apply updates for all npm lockfiles. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Reference: #3780 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Reference: #3102 Reference: #1514 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Reference: #3746 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne

Thanks... here are some first nits.

Adds is_direct attribute to differentiate between direct dependecy relationships and dependencies listed in lockfiles which have both direct and transitive dependencies together, which will have is_direct as False. Reference: #3780 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne

Some final nits... then please merge!

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra · 2024-06-17T13:54:03Z

@pombredanne Thanks for your comments, suggestions and improved texts for the attributes. I've added all of them, and now ready to merge.

Fixes a bug where there was remaining double quotes on namespace and name for dependencies, which was effecting package resolution. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne

Good to merge with just a minor nit wrt. a function name!

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra · 2024-06-19T14:18:42Z

All green, merging! Thanks @pombredanne

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 27e5537 to 69bb143 Compare May 16, 2024 14:54

Add basic pnpm lockfile parsers

ccf346a

Add parsers for pnpm-lock.yaml v5 and v6, and shrinkwrap.yaml specs with examples and package assembly. Reference: #3766 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 69bb143 to ccf346a Compare May 22, 2024 10:47

Update is_resolved from resolved_packages

514e1fd

Apply updates for all npm lockfiles. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 1869154 to b9eb01d Compare May 30, 2024 15:19

AyanSinhaMahapatra added 3 commits June 4, 2024 15:17

Update resolved package support for npm #3780

86d9d63

Reference: #3780 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Add attribute to track private packages #3102

f848741

Reference: #3102 Reference: #1514 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Add npm and pnpm workspace example #3746

ae2af48

Reference: #3746 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch from b9eb01d to f055ea7 Compare June 4, 2024 09:48

Add npm and pnpm workspace support #3746

69ea6b7

Reference: #3746 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch 2 times, most recently from 02e83da to 3e11778 Compare June 10, 2024 10:02

pombredanne changed the title ~~Add basic pnpm support and improve npm support~~ Resolve dependencies and improve JS support Jun 10, 2024

pombredanne requested changes Jun 10, 2024

View reviewed changes

Comment thread src/packagedcode/models.py Outdated

Comment thread src/packagedcode/npm.py Outdated

Comment thread src/packagedcode/npm.py Outdated

Comment thread src/packagedcode/npm.py

Comment thread src/packagedcode/npm.py Outdated

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 3e11778 to 448fc36 Compare June 10, 2024 14:28

Add is_virtual attribute and update tests

9af1547

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch 2 times, most recently from 74f402d to 82a51c0 Compare June 13, 2024 10:03

Merge branch 'develop' into improve-npm-support

b24b29f

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 82a51c0 to b24b29f Compare June 13, 2024 10:57

AyanSinhaMahapatra requested a review from pombredanne June 14, 2024 16:32

Handle duplicate aliases in yarn v1

ce8e0d1

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne requested changes Jun 17, 2024

View reviewed changes

Address feedback and comments

f74a89b

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the improve-npm-support branch from 20420b2 to f74a89b Compare June 17, 2024 13:50

AyanSinhaMahapatra requested a review from pombredanne June 17, 2024 13:54

Fix yarn lock v1 parser bug extra quotes

ae22b45

Fixes a bug where there was remaining double quotes on namespace and name for dependencies, which was effecting package resolution. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne reviewed Jun 19, 2024

View reviewed changes

Comment thread src/packagedcode/utils.py Outdated

pombredanne approved these changes Jun 19, 2024

View reviewed changes

Comment thread src/packagedcode/npm.py Outdated

Support resolutions in npm package.json

5536ede

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra mentioned this pull request Jun 19, 2024

Determine whether a package is present in a codebase #3811

Open

Address feedback and update CHNAGELOG and docs

8f932da

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra merged commit 2295c7f into develop Jun 19, 2024

AyanSinhaMahapatra deleted the improve-npm-support branch June 19, 2024 14:18

Uh oh!

Uh oh!

Conversation

AyanSinhaMahapatra commented May 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AyanSinhaMahapatra commented Jun 17, 2024

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AyanSinhaMahapatra commented Jun 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AyanSinhaMahapatra commented May 16, 2024 •

edited

Loading