Show the Layer information in scan results

[sameer1046]

Currently the output showing all components found in the tar file.
Expectation:
The output should show the Layer and the packages in that layer.
E.g. Just like Tern Tool
Layer 1:
info: Found 'Alpine Linux v3.10' in /etc/os-release.
Packages found in Layer1: musl-1.1.19-r10, busybox-1.28.4-r3, alpine-baselayout-3.1.0-r0, alpine-keys-2.1-r1, libressl2.7-libcrypto-2.7.5-r0, libressl2.7-libssl-2.7.5-r0, libressl2.7-libtls-2.7.5-r0, ssl_client-1.28.4-r3, zlib-1.2.11-r1, apk-tools-2.10.1-r0, scanelf-1.2.3-r0, musl-utils-1.1.19-r10, libc-utils-0.7.1-r0

[aalexanderr]

Duplicates #5

[pombredanne]

Merged from "Track which Docker image/layer a resource or package is found #5"

pombredanne commented on Sep 15, 2020

In a docker analysis pipeline, we have the layer information as this is always part of the resource path. We may also have the image name in the path, but that's not guaranteed. We need to have a way to attach explicitly to a discovered package and a codebase resource which docker image it is found in and also determine what is the base image/base image layers.
In the simplest way, this would be stored as extra_data attached to each of these objects and extra_data should also be returned with the JSON results.

[pombredanne]

@sameer1046 you wrote:

Currently the output showing all components found in the tar file.

Actually this is what happens in the standard "docker" pipeline at https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/docker.py

1. for a given image tarball, one or more "Image" is created, one for each configuration JSON found in the manifest.json
2. then each layer is extracted separately in a directory named after its id. (Note that we could squash the layers in a single rootfs alternatively using this https://github.com/nexB/container-inspector/blob/main/src/container_inspector/rootfs.py#L38 )
3. on each Image, a "Distro" is created using heuristics mostly based on the /etc/os-release file in the first layer at least on Linux. The upcoming Windows support has no os-release file and will use a different approach.
4. extra image data (mostly from the manifest) are attached to the project
5. we collect a resource inventory of all files across images and layers
6. we then detect system packages starting from the bottom layer and progressing upwards. If a package/version is present in the package DB of a layer N and also in N+1, then the N+1one is ignore (since it really does not exists in N+1, but only in N where it was first installed) See https://github.com/nexB/container-inspector/blob/e688750ba5533dcad53f456328e5fab69904a580/src/container_inspector/image.py
7. then we do further scans for application packages and files (each time working on the remaining files not part of a system then application package) and so on.

Therefore we know in the DB which layer a package is in, and this could be made more explicit

Let's design how to report it!

• We could find a way to report pipeline-specific extra data in the standard report and JSON, OR,
• We create a pipeline step to the Docker pipeline that will create a Docker-specific report that is layer-aware.

The second option feels better and would allow rapid evolution of the report format without impacting the standard reporting for all other pipelines.

[pombredanne]

@maxhbr FYI

[tdruez]

#353 merged.