Follow references to license in another file: ScanCode reports the "SEE LICENSE IN <filename>" text in an NPM package.json as "Unkown"

[sschuberth]

For non-SPDX / propretary licenses, NPM suggests to use a value of "SEE LICENSE IN " for the "license" key in package.json. If the license file is called LICENSE, that text ends up to be "SEE LICENSE IN LICENSE". At least ScanCode 2.9.7 reports this string as an "Unknown" license when scanning package.json. IMO, that text should not trigger a license finding in this case.

[sschuberth]

/cc @tsteenbe

[pombredanne]

@sschuberth

IMO, that text should not trigger a license finding in this case.

I think it should report a license and yes, unknown is not great.
It would be a welcomed addition to effectively deference these "see license" type of references.

I have some plans to actually dereference any such detection in npm and beyond.

For a start, license detection rules have a new field called "referenced_filenames" which is a list of filename references. See https://github.com/nexB/scancode-toolkit/search?l=YAML&q=%22referenced_filenames%22

Based on that, the logic will then be something more or less this way:
If a license is detected and contains "referenced_filenames", check if the filename exists uniquely in the same directory or in a parent/ancestor directory. If found, report the license detected in that "see file" in the current file as a new match with an appropriate type, (and possibly remove the "see file" match if it pointed to an unknown license.)

For special cases such as npm "license" attribute that use a special convention, the logic would be to parse that directly and inject that logic in the code https://github.com/nexB/scancode-toolkit/blob/fd0a95a04658178b8b4e74351bb84a392b618383/src/packagedcode/licensing.py#L68 that does declared license normalization.

[sschuberth]

I have some plans to actually dereference any such detection in npm and beyond.

I was briefly thinking about something like that, too, but felt it was overkill in this particular case as usually not only package.json is scanned, but all files in the package's repository / artifact, which includes the referenced file, and hence the license in there would be picked up anyway.

[pombredanne]

At this stage most license rules have been tagged with a referenced_filenames if they reference such a file. The next step is going to be to follow the referenced filenames in the general cases and in the special case of npm "SEE LICENSE IN" and point to the referenced license instead.

[AyanSinhaMahapatra]

@akugarg Here's an example npm package, see https://github.com/mongodb-js/vscode/blob/master/package.json#L34.

Here "license": "SEE LICENSE IN LICENSE.txt", references this file https://github.com/mongodb-js/vscode/blob/master/LICENSE.txt. You can find more here

We have to create a post-scan plugin with the following steps:

1. first find all codebase resources with a rule which has a referenced filename.
2. get rid of some cases of referenced filenames (some debian references beginning with usr/share/common-licenses/ will not be found in the directory tree.)
3. try to find the file in the referenced-filename first in the same directory, if not then in the codebase, and do nothing if not found.
4. If found, update the unknown license match found before, with the license matches for the file (and add a flag maybe).
5. test this with examples in multiple npm packages (all cases same directory/other directory/do nothing), where this is found a lot. You can just copy the two files mentioned above for a unittest.

[AyanSinhaMahapatra]

Btw, instead of a post-scan plugin, this could be a process_codebase step in a scan plugin, similar to here.

The get_scanner function for the license plugin is already there. The process_codebase could be defined inside the license plugin as a post processing step, but not a post-scan plugin explicitly.

Also, we can focus on adding this just for files in the same directory, with proper tests, and then move on to adding more complicated cases in packagedcode.

[akugarg]

From chat@gitter

Akanksha:
For the reference part the step which is done here akugarg@1a7ea7d#diff-9b8c9108cb6f529dc48ef0aba7a714c7e2b53feac33c7dc91eac423685a194b7R185 it will be present here only right since for every match we need to check for referenced files?

Ayan:
Why is the check done here, we can do this is the process_codebase function in plugin_license right, like we discussed previous day? The entire code can live there, unless I'm missing something?
we have access to the license list and LicenseMatch objects in it already in the process_codebase so why not do it there entirely.
Yes but this get_licenses is basically an API, which populates each resource with what is returned from the get_scanner function, so here the licenses list is added as the resource_attribute for each resource. Now that is available to all process_codebase functions that is called in either the same scan plugin/ scan plugins run after that, or post scan plugins. So we need not modify it directly at get_licenses but later in a seperate function which is the process_codebase function.
I.e. yes you can do what you are doing, it would obviously work and do what it is supposed to do. But it will change the functionality of get_licenses. We don't want that as it is used as an API in many other places widely (inside and outside scancode-toolkit). We want this to be added to the plugin_license post-processing step instead, so this step will only be done when the plugin is called in scancode-toolkit directly, and not via the API.

Akanksha:
Also one point how we will we have the access to "matches" thing since we would need to get "match.rule.referenced_filenames" ?

[AyanSinhaMahapatra]

See how we get resources here and from them resource attributes are accessed here.

[akugarg]

akugarg@8681972
I tried this but how to get root_path for particular match since we need this for searching in particular directory.

[AyanSinhaMahapatra]

See here for functions realted to Resource object on how to get parent resource. Have a look at the other functions for this class too.

[pombredanne]

This should be still open:

1. the npm part is not implemented
2. there are issue wrt. adding license matches of referenced files to the main license match with "SEE LICENSE IN"...

• for now the matches are just copied
• there is no indication (attribute, or else) that tells me that a match is derived from following the license of a referenced filename. There is no indication either that the referenced filename matches are now copied/added to the referencing place
• how we should handle these is not entirely clear to me:
  • should these be matches side by side like now?
  • or sub-matches added as an attribute to the referencing match?
  • should the referenced matches be moved rather than copied in the referencing?
  • how to track which files/path a referenced match is for (this is lots for now)? (and then us that for display of matched text?)
  • should the referencing match be removed now that we have referenced matches for it?
  • should the referenced matches be follow even if the match has a low coverage (and possibly the parts about the file references were not matched?)