Skip to content

Restructure pipelines for verbosity - #1074

Merged
tdruez merged 5 commits into
mainfrom
restructure-pipelines
Feb 14, 2024
Merged

Restructure pipelines for verbosity#1074
tdruez merged 5 commits into
mainfrom
restructure-pipelines

Conversation

@AyanSinhaMahapatra

@AyanSinhaMahapatra AyanSinhaMahapatra commented Feb 9, 2024

Copy link
Copy Markdown
Member

Remove scan_codebase_packages pipeline, and restructure inspect_packages pipeline into load_sbom and resolve_packages pipelines.

Reference: #1040
Reference: #1035
Reference: #1034
Reference: #1044

Remove scan_codebase_packages pipeline, and restructure inspect_packages
pipeline into load_sbom and resolve_packages pipelines.

Reference: #1035
Reference: #1034
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AyanSinhaMahapatra see my various comments.

Also, the following is missing:

  • Unit tests
  • Release notes (very important considering the major pipeline renaming, changes, and additions)
  • The Built-in pipe docs need to be updated with those changes

Comment thread scanpipe/pipelines/load_sbom.py Outdated
Comment thread scanpipe/pipelines/load_sbom.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Reference: #1074
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra

Copy link
Copy Markdown
Member Author

@tdruez thanks for the review comments!
I've updated the PR to address all the issues mentioned above.
Didn't update the docs/changelog before as I wanted you and @pombredanne to check out the pipeline restructuring and approve, but I've updated them now, and this is ready for another round of review. Thanks!

@AyanSinhaMahapatra AyanSinhaMahapatra linked an issue Feb 12, 2024 that may be closed by this pull request
@AyanSinhaMahapatra

Copy link
Copy Markdown
Member Author

See also comment at #1035 (reply in thread), we've addressed all the comments there and are adhering to the naming convention now for all pipelines.

@tdruez

tdruez commented Feb 12, 2024

Copy link
Copy Markdown
Contributor

@AyanSinhaMahapatra Thanks for the changes. Everything looks fine on my side but I'd like to get @pombredanne review as well before merging as this is quite a big change:

  • 2 new pipelines
  • 1 removed pipeline

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
I posted a few nits regarding wordsmithing.
Please review, apply the good ones and this is good to merge.

Comment thread scanpipe/pipelines/load_sbom.py Outdated
Comment thread scanpipe/pipelines/load_sbom.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/resolve_dependencies.py Outdated
Comment thread scanpipe/pipelines/scan_codebase.py Outdated
Comment thread scanpipe/pipelines/scan_single_package.py Outdated
@AyanSinhaMahapatra
AyanSinhaMahapatra force-pushed the restructure-pipelines branch 2 times, most recently from 51424c1 to e9ebbd4 Compare February 13, 2024 17:04
Suggested-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra linked an issue Feb 13, 2024 that may be closed by this pull request
@AyanSinhaMahapatra

Copy link
Copy Markdown
Member Author

@pombredanne thanks for the suggestions, I've added them all with some more docstring updates!

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.... just one last nit, but nothing in the way of merging.


.. _pipeline_scan_codebase_package:

Scan Codebase Package

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is renamed... we may still need an entry in the doc? This can be added later.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pombredanne we do have the new pipelines in the docs correctly, see: https://github.com/nexB/scancode.io/blob/restructure-pipelines/docs/built-in-pipelines.rst?plain=1#L61

Is this what you meant?

We are not having entries on renamed pipelines in the docs anymore: see https://github.com/nexB/scancode.io/pull/1053/files#diff-8a1644efb2ff73c9d7a85c18c73d2558528056d2fe7bb2adafe2502df72f4e2e also. The old pipelines are available only through the API/CLI for compatibility where they are renamed to the new pipelines. Otherwise we are not having the old pipeline names anywhere.

@tdruez
tdruez merged commit 7bb4499 into main Feb 14, 2024
@tdruez
tdruez deleted the restructure-pipelines branch February 14, 2024 07:54
@tdruez

tdruez commented Feb 14, 2024

Copy link
Copy Markdown
Contributor

@AyanSinhaMahapatra Merged, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rename pipelines Inspect_manifest pipeline error for get_manifest_inputs()

3 participants