The new `from_file` field contains the name of the input directory even with `--strip-root`

[sschuberth]

Description

I'd expect the value of the new from_file field inside the JSON's matches object to be a path relative to the input directory (if --strip-root is given). Instead, it is a relative path, but it contains the name of the input directory.

How To Reproduce

Using ScanCode 32.1.0, run a scan on the files mentioned at #3648. This gets you

  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "32.1.0",
      "options": {
        "input": [
          "/home/sebastian/Downloads/files"
        ],
        "--copyright": true,
        "--info": true,
        "--json-pp": "scancode-result.json",
        "--license": true,
        "--strip-root": true,
        "--timeout": "300.0"
      },

  "files": [
    {
      "path": "COPYING",
      "type": "file",
      "name": "COPYING",
      "base_name": "COPYING",
      "extension": "",
      "size": 2775,
      "date": "2024-01-29",
      "sha1": "66933e63e70616b43f1dc60340491f8e050eedfd",
      "md5": "97d554a32881fee0aa283d96e47cb24a",
      "sha256": "bcb02973ef6e87ea73d331b3a80df7748407f17efdb784b61b47e0e610d3bb5c",
      "mime_type": "text/plain",
      "file_type": "ASCII text",
      "programming_language": null,
      "is_binary": false,
      "is_text": true,
      "is_archive": false,
      "is_media": false,
      "is_source": false,
      "is_script": false,
      "detected_license_expression": "public-domain AND lgpl-2.1-plus AND gpl-2.0-plus AND (public-domain AND gpl-2.0-plus AND gpl-3.0-plus) AND (other-permissive AND other-copyleft) AND public-domain-disclaimer AND lgpl-2.1",
      "detected_license_expression_spdx": "LicenseRef-scancode-public-domain AND LGPL-2.1-or-later AND GPL-2.0-or-later AND (LicenseRef-scancode-public-domain AND GPL-2.0-or-later AND GPL-3.0-or-later) AND (LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft) AND LicenseRef-scancode-public-domain-disclaimer AND LGPL-2.1-only",
      "license_detections": [
        {
          "license_expression": "public-domain AND lgpl-2.1-plus AND gpl-2.0-plus AND (public-domain AND gpl-2.0-plus AND gpl-3.0-plus) AND (other-permissive AND other-copyleft) AND public-domain-disclaimer AND lgpl-2.1",
          "license_expression_spdx": "LicenseRef-scancode-public-domain AND LGPL-2.1-or-later AND GPL-2.0-or-later AND (LicenseRef-scancode-public-domain AND GPL-2.0-or-later AND GPL-3.0-or-later) AND (LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft) AND LicenseRef-scancode-public-domain-disclaimer AND LGPL-2.1-only",
          "matches": [
            {
              "license_expression": "public-domain",
              "spdx_license_expression": "LicenseRef-scancode-public-domain",
              "from_file": "files/COPYING",

So compare

"path": "COPYING",

to

"from_file": "files/COPYING",

which makes it unnecessary hard to check whether a given finding is actually just a reference or not by comparing whether both fields point to the same path.

I propose to either:

• apply a fix for both fields to contain comparable paths,
• only add from_file at all if it's pointing to a different file than path,
• add a dedicated is_reference field to yet more easily filter out findings that are just references to other scanned files.

System configuration

• What OS are you running on? Fedora Linux 39
• What version of scancode-toolkit was used to generate the scan file? 32.1.0
• What installation method was used to install/run scancode? pip

[AyanSinhaMahapatra]

@sschuberth thanks for the bug report!

apply a fix for both fields to contain comparable paths,

I think this would be the cleanest approach here, since we have already considered implementing the other two options you mentioned:

• only add from_file at all if it's pointing to a different file then path,
• add a dedicated is_reference field to yet more easily filter out findings that are just references to other scanned files.

And after a discussion with @pombredanne decided to implement the from_file attribute the way it is today, every match is populated with the file path it originated from, regardless of it belonging to the current file or a different file.

[pombredanne]

@sschuberth In hindsight, I have always had reservation wrt. the --strip-root option, and I wonder if this should not be removed entirely

[sschuberth]

Please keep in mind that when sharing scan results it is actually convenient to have only relative paths in the JSON (to make results somewhat "relocatable"). While you probably could make them relative WRT headers.options.input, users might also feel uncomfortable with disclosing the directory structure where they store source code, as it could reveal things about the origin in the names.

So, personally I'm not against --strip-root, but then the behavior should change so as if --strip-root was always specified, and always only relative paths should be used, IMO.

[sschuberth]

Hmm, looks like ScanCode 32.2.0 changed the behavior to the worse: Running

scancode --copyright --info --json-pp scancode.json --license --license-references --processes 3 --strip-root --timeout 0.05 .

on https://github.com/jshttp/mime-types/tree/2.1.18 gives

      "options": {
        "input": [
          "."
        ],
        "--copyright": true,
        "--info": true,
        "--json-pp": "scancode.json",
        "--license": true,
        "--license-references": true,
        "--processes": "3",
        "--strip-root": true,
        "--timeout": "0.05"
      },

and entries like

      "reference_matches": [
        {
          "license_expression": "mit",
          "license_expression_spdx": "MIT",
          "from_file": "mime-types/LICENSE",
          "start_line": 1,
          "end_line": 1,
          "matcher": "2-aho",
          "score": 100.0,
          "matched_length": 3,
          "match_coverage": 100.0,
          "rule_relevance": 100,
          "rule_identifier": "mit_27.RULE",
          "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_27.RULE"
        }
      ]

which means that ORT's fixup logic to strip headers.options.input from from_file in order to get proper relative paths does no work anymore.