C source code line mistaken for MPL licence

[Thalley]

Description

A C source code line is mistaken for the MPL 1.1 License. The line of code is

SHELL_CMD_ARG_REGISTER(mpl, &mpl_cmds, " ", cmd_mpl, 1, 1);

where mpl, 1, 1 is seemingly read as the MPL 1.1. license, and SHELL_CMD_ARG_REGISTER is a macro

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Linux
• What version of scancode-toolkit was used to generate the scan file? 3.1.1

[pombredanne]

@Thalley Thanks... that's a good (and rare) catch and your analysis is correct: this is matched by this license rule: https://github.com/nexB/scancode-toolkit/blob/abfe573bfe1f8a18c17418a45e0c1ff57c8a07fc/src/licensedcode/data/rules/spdx_license_id_mpl-1.1_for_mpl-1.1.RULE#L1 with contains these words only mpl-1.1 which is the ID for the MPL 1.1 license.
ScanCode ignores all punctuation except trailing + in a word, so mpl, 1, 1 is matched.
The simple way around this will be to create a new rule that is tagged as a is_false_positive: yes and with this text cmd_mpl, 1, 1 such that if mpl 1 1 is detected this will not be reported as a detection when prefixed with the word cmd

For reference the scan I ran was:

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "3.2.2rc3",
      "options": {
        "input": [
          "foo"
        ],
        "--json-pp": "-",
        "--license": true,
        "--license-text": true,
        "--license-text-diagnostics": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2020-10-27T144655.632241",
      "end_timestamp": "2020-10-27T144730.823424",
      "duration": 35.19121718406677,
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "foo",
      "type": "file",
      "licenses": [
        {
          "key": "mpl-1.1",
          "score": 50.0,
          "name": "Mozilla Public License 1.1",
          "short_name": "MPL 1.1",
          "category": "Copyleft Limited",
          "is_exception": false,
          "owner": "Mozilla",
          "homepage_url": "http://www.mozilla.org/MPL/MPL-1.1.html",
          "text_url": "http://www.mozilla.com/MPL/1.1/index.html",
          "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:mpl-1.1",
          "spdx_license_key": "MPL-1.1",
          "spdx_url": "https://spdx.org/licenses/MPL-1.1",
          "start_line": 1,
          "end_line": 1,
          "matched_rule": {
            "identifier": "spdx_license_id_mpl-1.1_for_mpl-1.1.RULE",
            "license_expression": "mpl-1.1",
            "licenses": [
              "mpl-1.1"
            ],
            "is_license_text": false,
            "is_license_notice": false,
            "is_license_reference": true,
            "is_license_tag": false,
            "matcher": "2-aho",
            "rule_length": 3,
            "matched_length": 3,
            "match_coverage": 100.0,
            "rule_relevance": 50.0
          },
          "matched_text": "mpl, 1, 1);"
        }
      ],
      "license_expressions": [
        "mpl-1.1"
      ],
      "percentage_of_license_text": 27.27,
      "scan_errors": []
    }
  ]
}

[Thalley]

@pombredanne Thanks for the quick reply - I'll try applying your suggestion. Is there any way the problem can be solved at the root (i.e. in scancode), rather than circumvented at the users?

[pombredanne]

@Thalley this is going to be fixed in scancode alright! I was just hinting on how to solve it in scancode. If you want to submit a PR that would be much welcomed.
Here is what this would look like:

1. add a test:

• in scancode-toolkit/tests/licensedcode/data/licenses create a new directory 2304-no-mpl
• in this directory add a file named shell.c with this content:
  SHELL_CMD_ARG_REGISTER(mpl, &mpl_cmds, " ", cmd_mpl, 1, 1);
• in this directory add a file named shell.c.yml with this content:
  notes: This test should not detect anything as reported in ticket #2304
  (NB: the file names you pick do not matter BUT you need to have a .yml file with the same base name as your test file and here we expect NO license license to be detected therefore there is only a license)
• you can then run that one test with pytest -vvs --test-suite=all tests/licensecode -k 2304

2. add a rule to fix this:

• in scancode-toolkit/src/licensedcode/data/rules create a new file named not-a-license_mpl_in_c.RULE with this content:
  cmd mpl 1 1
• in scancode-toolkit/src/licensedcode/data/rules create a new file named not-a-license_mpl_in_c.yml with this content:

is_false_positive: yes
notes: this is not an MPL license. Found in a C macro in zephyr

and that's it: this will ensure that this is both tested and fixed

[pombredanne]

@Thalley that's if you care to send a PR... do not feel obligated!

[Thalley]

@pombredanne Thanks - Not sure if I'll get around to do that in the coming time as I have other higher priority matters to attend to. But I'll try to keep this issue in mind :)

[pombredanne]

@Thalley no problem, I will apply the proposed fix directly then.

[Thalley]

@pombredanne I was actually wondering why scancode was even looking for license in that particular line of code. Wouldn't any license information either be in a comment (in source code files), or in a text (.txt, .md, .rst, etc.) file?

It ought to be possible to, for known file type extensions like .c and .h, to only look for licenses inside commented sections (enclosed by /* */ or //).

I suppose that if all none-comment code could be excluded from the license checks, that might also speed up the scan.

[pombredanne]

@Thalley re:

I was actually wondering why scancode was even looking for license in that particular line of code. Wouldn't any license information either be in a comment (in source code files), or in a text (.txt, .md, .rst, etc.) file?

The approach is a comprehensive "no license left behind".

scancode is pretty dumb there and does not skip any file and does not even look at whether a file is code or not code and definitely not if this is code comment or code. It even looks in binaries alright after extracting strings (including wide strings from windows EXEs) and more often than not that quite useful and there are licenses and copyrights in compiled binaries.

It ought to be possible to, for known file type extensions like .c and .h, to only look for licenses inside commented sections (enclosed by /* */ or //).

In theory yes, but the issues comes with special cases
__license__ variable in Python. MODULE_LICENSE in C/Linux LKM, etc.
So you would need to distinguish between comments and code; and in code between symbols and literals where some symbols matter and some do not.
The cost of doing all this lexing and parsing is high so it would need to be optimized in Cython, C/C++ or Rust or Go... e.g. some native code.
In fact I did use the Pygments lexers just for that purpose but that was long time ago. The lexers had to be streamlined for the most popular programming languages to speed things up a bit. The gain in accuracy were rather minor but the cost of doing all that lexing was high performance-wise and that was a also a lot of code to maintain.

I suppose that if all none-comment code could be excluded from the license checks, that might also speed up the scan.

That would be worth trying again for sure, but my past, outdated experience was that it did not help at all with speed...

With all that said, I truly think it would be worth experimenting with this and having this as "lighter weight" "surface" license scan where you would combine:

1. looking only at text files
2. in code file, looking only at comments.

This would then likely be faster, and even if it would fail several of the tests we have I can see the benefits of a "quick scan" approach as an option.

As it happens there is a library at Fossoloy that does comment vs. code splitting at https://github.com/fossology/Nirjas/ so we could try it out (by @hastagAB during his GSoC project for Fossology)

[mjherzog]

I think that an option to look only at Comments in source code would be worthwhile. The cases where we are looking for things like MODULE_LICENSE seem to be pretty specific and I would hope/expect that there are not a lot of them in practical usage.

[mjherzog]

A license detection within code is useful, but it should be somehow delineated from a license detection within comments because the former is never a sufficient license assertion. For example if you have a license within a MODULE_LICENSE( ) statement that would not on its own be sufficient as a clear license assertion. So there should be some way to document when a license detection comes from code rather than comments.

[mjherzog]

For special cases like MODULE_LICENSE it would probably be best to use a separate ScanCode plugin that you only run when your codebase fits that use case - i.e. no reason to look for MODULE_LICENSE in anything but C/C++ code where the product codebase includes Linux drivers or similar.

[Thalley]

@pombredanne @mjherzog Any updates on this?

[pombredanne]

@Thalley we have some design in progress for a solution in #2403
In the meantime I am just pushing the rule I mentioned above... though I could not find an exact pointer to the code in question.
Do you have a URL by chance?