support `inspect-manifest` for CycloneDx 1.4 by keshav-space · Pull Request #592 · aboutcode-org/scancode.io

keshav-space · 2023-01-27T15:44:59Z

A CycloneDx component can have a list of components, those are dumped to extra_data as nestedComponents. Furthermore, these lists of components are recursively parsed and treated as normal package.
The Component may have multiple URLs in externalReferences. The first URL of the reference is added to the applicable package_data URL, while the rest are dumped in extra_data as externalReferences.

fixes #583

Signed-off-by: Keshav Priyadarshi git@keshav.space

AyanSinhaMahapatra

Thanks! A few initial nits and comments for you.

AyanSinhaMahapatra · 2023-01-31T14:41:21Z

+    Requires the `jsonschema` library.
+    """
+    try:
+        import jsonschema


Isn't jsonschema in install_requires? Will this ever fail?

Isn't jsonschema in install_requires?

I was just being consistent with spdx.validate_document 😅
https://github.com/nexB/scancode.io/blob/0b4053ebe8c229c18308c0e96af8346842b75a7b/scanpipe/spdx/__init__.py#L648-L659

Will this ever fail?

Let me confirm if it!

Will this ever fail?

I checked, it will never fail. Will get this removed.

Maybe we can remove this in spdx too (?)

Maybe we can remove this in spdx too (?)

I will enter an issue for that and if there is no objection then we can go ahead and remove it.

Sounds good?

This is just a tiny nit, too small for a separate issue, just keep this conversation open for others to comment.

AyanSinhaMahapatra · 2023-02-01T16:45:44Z

@kpriyadarshi I ran a inspect_manifest pipeline from your branch and it worked great!

INFO "GET /project/add/ HTTP/1.1" 200 35090
INFO Fetching "https://raw.githubusercontent.com/keshav-space/scancode.io/import_cyclonedx/scanpipe/tests/data/cyclonedx/asgiref-3.3.0.bom.json" using fetch_http
INFO Run[e650ea41-ba7a-4e61-85a5-5abfd6f316a7] Enter `execute_pipeline_task` Run.pk/Task.id=e650ea41-ba7a-4e61-85a5-5abfd6f316a7
INFO Run[e650ea41-ba7a-4e61-85a5-5abfd6f316a7] Run pipeline: "inspect_manifest" on project: "cyclonedx-test-asgiref-3.3.0"
INFO 2023-02-01 16:39:52.46 Pipeline [inspect_manifest] starting
INFO 2023-02-01 16:39:52.46 Step [get_manifest_inputs] starting
INFO 2023-02-01 16:39:52.46 Step [get_manifest_inputs] completed in 0.00 seconds
INFO 2023-02-01 16:39:52.46 Step [create_packages_from_manifest] starting
INFO 2023-02-01 16:39:52.46 `cyclonedx` input detected for /home/ayansinha/nexB/main_repos/gh-cli-repos/scancode.io/var/projects/cyclonedx-test-asgiref-330-9e28864f/input/asgiref-3.3.0.bom.json
INFO 2023-02-01 16:39:52.52 Step [create_packages_from_manifest] completed in 0.05 seconds
INFO 2023-02-01 16:39:52.52 Pipeline completed
INFO Run[e650ea41-ba7a-4e61-85a5-5abfd6f316a7] Update Run instance with exitcode, output, and end_date
INFO "POST /project/add/ HTTP/1.1" 302 0
INFO "GET /project/9e28864f-d580-4e10-8ec0-bb515c5d66d5/ HTTP/1.1" 200 19754
INFO "GET /project/9e28864f-d580-4e10-8ec0-bb515c5d66d5/charts/ HTTP/1.1" 200 4491
INFO "GET /project/9e28864f-d580-4e10-8ec0-bb515c5d66d5/packages/ HTTP/1.1" 200 8867
INFO "GET /project/9e28864f-d580-4e10-8ec0-bb515c5d66d5/packages/30/ HTTP/1.1" 200 22602
INFO "GET /project/add/ HTTP/1.1" 200 35090

The license_expression also seems to work fine:

License expression

    bsd-new AND bsd-new

Declared license

    BSD-3-Clause AND BSD-3-Clause

I'll check with some more complicated cyclonedx boms, and let you know!

tdruez

@keshav-space great start, here's some suggestions to improve the code:

We do not use typing in ScanCode.io at the moment, you want to be consistent with the existing codebase convention.
Do not use the or [] syntax.
Do not use obj nor iterableas variable names but try to name things what they are.
Some docstring are not ending with a .
The integraiton test is great, but we are missing a unit test for each functions of the cyclonedx module.

tdruez

@keshav-space see my various comments for refinements, also:

You have "CycloneDx" and "CYCLONEDX" occurrences that need to be replaced by the proper syntax: "CycloneDX".
I gave you "write" access on the repo so in the future you can create branches directly on the repo.

tdruez · 2023-02-22T10:35:08Z

+    resolver = jsonschema.RefResolver(
+        base_uri="file://" + str(pathlib.Path(__file__).parent), referrer=schema
+    )
+
+    validator = jsonschema.Draft7Validator(schema=schema, resolver=resolver)
+
+    validator.validate(instance=document)


Is this complexity needed?
The basic jsonschema.validate(instance=document, schema=schema) is not enough in the CycloneDX case?

jsonschema.validate(instance=document, schema=schema) will work, but if BOM contains an invalid SPDX license expression, then hoppr-cyclonedx-models won't be able to parse it. The same thing goes for JSF.

- CycloneDx `component` can have a `list of components`, those are dumped to extra_data as `nestedComponents`. Furthermore, these lists of components are recursively parsed and treated as normal package. - The Component may have multiple URLs in externalReferences. The first URL of the reference is added to the applicable package_data URL, while the rest are dumped in extra_data as externalReferences. Fixes aboutcode-org#583 Signed-off-by: Keshav Priyadarshi <git@keshav.space>

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

keshav-space marked this pull request as draft January 27, 2023 15:55

keshav-space marked this pull request as ready for review January 30, 2023 13:12

keshav-space requested review from AyanSinhaMahapatra, DennisClark, JonoYang, pombredanne and tdruez January 30, 2023 13:13

keshav-space force-pushed the import_cyclonedx branch 2 times, most recently from 1caeffc to 3e37835 Compare January 30, 2023 15:35

AyanSinhaMahapatra requested changes Jan 31, 2023

View reviewed changes

keshav-space requested a review from AyanSinhaMahapatra February 2, 2023 20:02

keshav-space force-pushed the import_cyclonedx branch from ed8e2ed to a496619 Compare February 2, 2023 20:04

keshav-space force-pushed the import_cyclonedx branch from a496619 to 685ef9d Compare February 15, 2023 11:50

tdruez requested changes Feb 15, 2023

View reviewed changes

keshav-space force-pushed the import_cyclonedx branch 2 times, most recently from 5ae8195 to 75ecd2a Compare February 20, 2023 16:17

keshav-space requested a review from tdruez February 20, 2023 16:54

tdruez requested changes Feb 22, 2023

View reviewed changes

keshav-space added 5 commits March 3, 2023 00:03

test cyclonedx inspect_manifest

55a8fac

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

add docstrings

2506a6a

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

Address suggestions

99c638a

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

Add unit test for cyclonedx

1f2c7ce

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

keshav-space force-pushed the import_cyclonedx branch from 75ecd2a to b78ee53 Compare March 2, 2023 18:38

Address suggestions

043bbb9

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

keshav-space force-pushed the import_cyclonedx branch from b78ee53 to 043bbb9 Compare March 2, 2023 18:42

keshav-space requested review from tdruez March 2, 2023 19:00

tdruez merged commit ae4860c into aboutcode-org:main Mar 3, 2023

keshav-space deleted the import_cyclonedx branch June 8, 2023 11:02

Uh oh!

Uh oh!

Conversation

keshav-space commented Jan 27, 2023

Uh oh!

AyanSinhaMahapatra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AyanSinhaMahapatra commented Feb 1, 2023

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants