RFC: The future of pipelines

[pombredanne]

We are facing some interesting growth pain around pipelines: how we name them, how we can rename and/or deprecate them, how we can compose and reuse them. This is a "meta" issue to track the problems, solution elements and relate to other tickets and discussions.

Here are some directions and discussions:

• on the naming of pipelines, see Define a modular naming convention for pipelines #1035 for a discussion
• on how we could compose pipelines, or run more than one: See RFC: Introduce pipeline templates #996

Some Problems:

• Some pipelines need renaming. See above and Rename pipelines #1044
• Some pipelines should be retired and are no longer needed: should we track deprecation? or just retire them?
• Most pipelines cannot be run together on a project. Flag "addon" pipelines Do not include "Add-on" pipelines in the "New project" form choices #1041
• Perhaps adding more there and providing a "help" icon would help. 2eb587d
• Refactor deploy_to_develop pipeline in tech-specific pipelines #1045 Split d2d pipeline in two: Java and JS. For instance, the deploy_to_development needs to be split into one pipeline for Java and one pipeline for JavaScript that can be used standalone. The matching part should also be its own pipeline. And we still want to have a combined Java/JavaScript.
• I want to have d2d pipeline capabilities for several discrete technologies/stacks: Java, or JS, or Python or NuGet such that I can focus this on a single package.
• I want also to be able to analyze large codebase with multiple technologies, providing my input on which techs are used: for instance, Java and JS, or PHP and JS
• Some pipeline descriptions are too terse and should explain more.
  Note that the full description and list of steps is listed on the popup.
• We could reorg the way we present pipelines in the UI to ease navigation and selection.
• The sort order of the pipelines is only alpha and is not the most obvious, most useful order
• We could add labels to each pipeline to help sort and filter them. We could tag older, legacy pipelines accordingly
• Some users are confused when selecting which pipeline to use and end up selecting the wrong or too many pipelines.
• It could be useful to have some arguments/option for a pipeline, like a general purpose d2d, where you set which tech flavor to use (java, js, C/C++). Add ability to "group" pipeline steps to control inclusion #1045 #1055

Some possible todos

• Create new "inspect_packages": Merge "inspect_manifest" with "scan_codebase_packages"
• Extract the python inspector in a new "resolve_dependencies"
• Create new "load_sbom" : extract the "SBOM" from "inspect_manifest" combine with "load_inventory". This could be only SBOMs: SPDX and CycloneDX
• ?? consider renaming load_inventory to "load_scan": This would be only our own scans (SCIO, SCTK, etc)
• Improve all pipelines documentation (description and details) with renaming in Rename pipelines #1044
• Refactor deploy_to_develop pipeline in tech-specific pipelines #1045 Add ability to "group" pipeline steps to control inclusion #1045 #1055
  • Create d2d analyze_deployed_java pipeline
  • Create d2d analyze_deployed_javascript pipeline
  • Create d2d match_code pipeline
• Create many new d2d pipelines: analyze_deployed_dotnet, php, python, native c/cpp, golang, rust,ruby, deb, rpm,...
• Create a analyze_deployed_code pipeline. Then select all the techs you care for using an arg/option Add ability to "group" pipeline steps to control inclusion #1045 #1055
• Have a way to scan to keep a stable order for pipelines display