Add Package parser for LKMs

[pombredanne]

LKMs binaries contain metadata (which are derived from the MODULE_* macros in the source code).
For instance:

license=GPL v2
description=Intel Broxton SoC pinctrl/GPIO driver
author=Mika Westerberg <mika.westerberg@linux.intel.com>
srcversion=BBE9DA7EDB097BBD2FA9FE4
alias=acpi*:INT34D1:*
alias=acpi*:INT3452:*
depends=pinctrl-intel
intree=Y
vermagic=4.4.0-51-generic SMP mod_unload modversions 

[pombredanne]

For reference, these are source-side parser: https://github.com/nexB/scancode-toolkit-contrib/blob/7c5c115a8c3144f60269cfaf2f59b1b639c120aa/src/sourcecode/kernel.py

[pombredanne]

See include/linux/module.h for details

[OctavioValdiviaMendoza]

I'm working on this with @diana-galeana

[OctavioValdiviaMendoza]

@pombredanne @AyanSinhaMahapatra Here is the implementation approach we are currently working on for LKM package parsing:

We added a DatafileHandler for compiled Linux Kernel Module files (.ko) in src/packagedcode/lkm.py. Instead of scanning the entire binary with regex, the handler opens the module as an ELF file, and locates the .modinfo section, from there it parses its NUL-terminated key=value entries. Following the metadata structure described in the issue and is generated from the MODULE_* macros found in include/linux/module.h.

The parsed metadata is stored as a list so that repeated fields like alias, author, firmware, and other keys are preserved. We currently normalize the module filename into the package name, map licenses, description, and version into standard PackageData fields, convert authors into Party Objects and preserve metadata that is not included in those normalized fields in extra_data.

For depends, the handler splits the comma-separates modules names and current represents each one as a DependentPackage at runtime. If an ELF files is without a .modinfo section or is invalid it would return no package data, and not entirely fail the scan.

Current test coverage includes .ko recognition, .modinfo extraction, handling of missing optional fields, repeated metadata, dependency parsing and normalization, multiple authors, full PackageData creation, preservation of unknown metadata, and invalid ELF handling.

Before finalizing the implementation, we would appreciate some guidance on a few points:

• is linux-kernel-module the preferred ScanCode package type?

• For entries such as depends=pinctrl-intel, should these be represented as DependentPackage objects, and is a purl such as pkg:linux-kernel-module/pinctrl-intel appropriate (our current implementation), or should the dependency remain unresolved without a purl?

• The current implementation uses pyelftools to read the ELF .modinfo section. I couldn't find any existing use or declared dependency for pyelftools in the repository, so I'd appreciate guidance on whether introducing it as a dependency is acceptable.

Looking forward to hearing your thoughts. Thank you!

[chinyeungli]

Thanks for hard work. A few thoughts on the points raised:

1. I believe .ko (LKMs) should be treated as file instead of package. That's an LKM is a compiled artifact instead of a standalone package ecosystem.

2. For dependencies, it should be treated as module/symbol level instead of package level. The depends metadata inside a .ko file refers to other .ko files/symbols, not package manager dependencies.

3. There is no linux-kernel-module package type. Following from (1) and (2), I don't think theses should be treated as package, but if we intended to treat these as packages, perhaps we should use pkg:generic/ instead.

4. Introducing pyelftools shouldn't be an issue as long as license compliance and dependency requirements are met and there are no known security vulnerabilities.

@pombredanne @AyanSinhaMahapatra input are welcome.

[mjherzog]

@chinyeungli Thank you

[OctavioValdiviaMendoza]

Thank you, @chinyeungli, for the feedback. The distinction that a .ko file is a compiled artifact rather than a package from a package-management ecosystem makes sense. I also understand that the .modinfo depends field contains the names of required kernel modules, not package-manager dependencies, so representing those entries as DependentPackage objects would be misleading.

Our implementation used the existing DatafileHandler/PackageData architecture because it appeared to be the closest existing ScanCode pattern for exposing structured metadata extracted from a binary. Based on your feedback, however, it sounds like this metadata should remain associated with the .ko file itself rather than being represented as a package.

If that is the intended direction, what would be the appropriate existing ScanCode architecture for exposing this metadata at the file level? Would a file-level ScanPlugin be the right approach?

For the .modinfo depends field, our proposed scope would be to preserve the required module names alongside the other extracted metadata. We would not attempt to locate the corresponding .ko files or model dependency relationships between them. Would that be the appropriate representation and scope?

We'd like to confirm the intended architecture, output structure, and scope before revising the implementation so that it aligns with the existing ScanCode design.

[chinyeungli]

I believe the file-level ScanPlugin and storing the depends as a simple list of required module strings without resolving them are the right approach. Looking forward to seeing the updated implementation!

[OctavioValdiviaMendoza]

Thank you, @chinyeungli, for confirming the direction. I’ve pushed the revised implementation.

The updated implementation:

• Replaces the DatafileHandler/PackageData approach with a file-level LinuxKernelModuleScanner subclass of ScanPlugin.
• Moves the implementation out of packagedcode to scancode.plugin_lkm.
• Adds an opt-in --lkm scan option and stores results in a linux_kernel_module attribute on the scanned file Resource.
• Preserves .modinfo fields as lists so repeated entries such as author, alias, and firmware are not lost.
• Stores depends as a simple list of required kernel-module names.
• Replaces the package-oriented tests with file-scanner and end-to-end CLI tests. The focused test suite passes with 8 tests.

I have two remaining questions about the location and names used in the implementation:

1. Is scancode.plugin_lkm the appropriate location for this structured file metadata?
2. Are --lkm and the linux_kernel_module Resource attribute appropriate public names?