JSON output from scancode toolkit doesn't work as the input for `gen`

[chinyeungli]

The CSV output from scancode toolkit usually contains duplicated resource rows as each detected license/copyright/url value will use one row which violate the aboutcode tk rule as no duplication is allowed.
On the other hand, the aboutcode tk should be able to take JSON as an input, but, for some reasons, it doesn't work with the current scancode's JSON output.

The main goal for this ticket is to ensure aboutcode tk (the gen option) works with the JSON output from scancode toolkit.

[mjherzog]

We should enhance AboutCode Toolkit to accept standard JSON output from ScanCode, but I changed this from a bug to an enhancement because AbC TK spec works with the current spec. This is a very important enhancement, but it will require significant design work for AbC Toolkit and likely for ABOUT files.

[chinyeungli]

I've looked into this and there are couple of issues that I can spot at the moment:

• about_resource is an essential key for the use of the aboutcode toolkit. The scancode output doesn't have the about_resource key but use the path instead. We have the utility transform to rename the column keys, but this only works for CSV, and not JSON file. In addition, we cannot simply replace path with the about_resource as for instance in a package scan, the scan may return information for path /code/project/package.json, but what we want the ABOUT file to reference is /code/project/ instead of /code/project/package.json

• Out current design will keep all the keys in the input meaning all the start_line, end_line detected in the scancode output will be kept in the ABOUT file which does not make sense. The only way to ignore these fields currently is by the transform utility with the column_filters:, but again, this utility will only work for CSV file.

• It doesn't make sense to include detected licese which have low license score in the ABOUT file.

• The current aboutcode toolkit can take simple JSON as an input such as

{
[{
		"path": "123",
		"type": "directory",
		"name": "123",
		"base_name": "123",
		"extension": "",
		"size": 0,
		"date": null,
		"sha1": null,
		"md5": null,
		"mime_type": null,
		"file_type": null,
		"programming_language": null,
		"is_binary": false,
		"is_text": false,
		"is_archive": false,
		"is_media": false,
		"is_source": false,
		"is_script": false,
		"licenses": [],
		"license_expressions": [],
		"holders": [],
		"copyrights": [],
		"authors": [],
		"packages": [],
		"emails": [],
		"urls": [],
		"files_count": 87,
		"dirs_count": 41,
		"size_count": 507355,
		"scan_errors": []
	}]
}

However, it doesn't work with the JSON output from scancode which the data is under the files such as

{
	"headers": [{
		"tool_name": "scancode-toolkit",
		"tool_version": "2.9.10",
		"options": {
			"input": "xxx",
			"--copyright": true,
			"--csv": "xxx.csv",
			"--email": true,
			"--info": true,
			"--json": "xxx.json",
			"--license": true,
			"--package": true,
			"--processes": "4",
			"--url": true
		},
		"notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
		"start_timestamp": "2019....",
		"end_timestamp": "2019....",
		"message": null,
		"errors": [],
		"extra_data": {
			"files_count": 87
		}
	}],
	"files": [{
		"path": "123",
		"type": "directory",
		"name": "123",
		"base_name": "123",
		"extension": "",
		"size": 0,
		"date": null,
		"sha1": null,
		"md5": null,
		"mime_type": null,
		"file_type": null,
		"programming_language": null,
		"is_binary": false,
		"is_text": false,
		"is_archive": false,
		"is_media": false,
		"is_source": false,
		"is_script": false,
		"licenses": [],
		"license_expressions": [],
		"holders": [],
		"copyrights": [],
		"authors": [],
		"packages": [],
		"emails": [],
		"urls": [],
		"files_count": 87,
		"dirs_count": 41,
		"size_count": 507355,
		"scan_errors": []
	}]
}

• Many fields (and custom fields) are treated and defined as single line item such as license_expressions in aboutcode toolkit. However, scancode treats these fields as an array which mean will cause problem in the generation.

[chinyeungli]

in another word, as @mjherzog mentioned, it will require significant design work for AbC Toolkit and likely for ABOUT files to have this work

[chinyeungli]

I've been working this to use scancode's scan as the input.
I am now able to use the json file from scancode's scan as the input for gen to generate ABOUT file
scancode's scan:

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "3.2.1rc2",
      "options": {
        "input": [
          "C:\\Users\\Desktop\\lic_scan"
        ],
        "--copyright": true,
        "--email": true,
        "--info": true,
        "--json-pp": "C:\\Users\\Desktop\\lic_scan\\test.json",
        "--license": true,
        "--package": true,
        "--url": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2020-11-10T020513.009859",
      "end_timestamp": "2020-11-10T020641.819564",
      "duration": 88.80970501899719,
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "lic_scan",
      "type": "directory",
      "name": "lic_scan",
      "base_name": "lic_scan",
      "extension": "",
      "size": 0,
      "date": null,
      "sha1": null,
      "md5": null,
      "sha256": null,
      "mime_type": null,
      "file_type": null,
      "programming_language": null,
      "is_binary": false,
      "is_text": false,
      "is_archive": false,
      "is_media": false,
      "is_source": false,
      "is_script": false,
      "licenses": [],
      "license_expressions": [],
      "percentage_of_license_text": 0,
      "copyrights": [],
      "holders": [],
      "authors": [],
      "packages": [],
      "emails": [],
      "urls": [],
      "files_count": 1,
      "dirs_count": 0,
      "size_count": 2011,
      "scan_errors": []
    },
    {
      "path": "lic_scan/lic.txt",
      "type": "file",
      "name": "lic.txt",
      "base_name": "lic",
      "extension": ".txt",
      "size": 2011,
      "date": "2020-11-09",
      "sha1": "fe4ff75272675c27279ddf101a2d6d7cd1e9ad64",
      "md5": "326d5674181c4bb210e424772c60fa80",
      "sha256": "3098be2717081cfe2b88a0faa221cb4f2326720cb8ce869199a9d6234f219319",
      "mime_type": "text/plain",
      "file_type": "ASCII text",
      "programming_language": null,
      "is_binary": false,
      "is_text": true,
      "is_archive": false,
      "is_media": false,
      "is_source": false,
      "is_script": false,
      "licenses": [
        {
          "key": "isc",
          "score": 98.29,
          "name": "ISC License",
          "short_name": "ISC License",
          "category": "Permissive",
          "is_exception": false,
          "owner": "ISC - Internet Systems Consortium",
          "homepage_url": "https://www.isc.org/software/license",
          "text_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style_with_legal_disclaimer_2",
          "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:isc",
          "spdx_license_key": "ISC",
          "spdx_url": "https://spdx.org/licenses/ISC",
          "start_line": 1,
          "end_line": 15,
          "matched_rule": {
            "identifier": "isc_48.RULE",
            "license_expression": "isc",
            "licenses": [
              "isc"
            ],
            "is_license_text": true,
            "is_license_notice": false,
            "is_license_reference": false,
            "is_license_tag": false,
            "matcher": "3-seq",
            "rule_length": 117,
            "matched_length": 115,
            "match_coverage": 98.29,
            "rule_relevance": 100.0
          }
        },
        {
          "key": "mit",
          "score": 100.0,
          "name": "MIT License",
          "short_name": "MIT License",
          "category": "Permissive",
          "is_exception": false,
          "owner": "MIT",
          "homepage_url": "http://opensource.org/licenses/mit-license.php",
          "text_url": "http://opensource.org/licenses/mit-license.php",
          "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:mit",
          "spdx_license_key": "MIT",
          "spdx_url": "https://spdx.org/licenses/MIT",
          "start_line": 19,
          "end_line": 20,
          "matched_rule": {
            "identifier": "mit_14.RULE",
            "license_expression": "mit",
            "licenses": [
              "mit"
            ],
            "is_license_text": false,
            "is_license_notice": false,
            "is_license_reference": true,
            "is_license_tag": false,
            "matcher": "2-aho",
            "rule_length": 2,
            "matched_length": 2,
            "match_coverage": 100.0,
            "rule_relevance": 100.0
          }
        },
        {
          "key": "mit",
          "score": 100.0,
          "name": "MIT License",
          "short_name": "MIT License",
          "category": "Permissive",
          "is_exception": false,
          "owner": "MIT",
          "homepage_url": "http://opensource.org/licenses/mit-license.php",
          "text_url": "http://opensource.org/licenses/mit-license.php",
          "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:mit",
          "spdx_license_key": "MIT",
          "spdx_url": "https://spdx.org/licenses/MIT",
          "start_line": 24,
          "end_line": 41,
          "matched_rule": {
            "identifier": "mit.LICENSE",
            "license_expression": "mit",
            "licenses": [
              "mit"
            ],
            "is_license_text": true,
            "is_license_notice": false,
            "is_license_reference": false,
            "is_license_tag": false,
            "matcher": "2-aho",
            "rule_length": 161,
            "matched_length": 161,
            "match_coverage": 100.0,
            "rule_relevance": 100
          }
        }
      ],
      "license_expressions": [
        "isc",
        "mit",
        "mit"
      ],
      "percentage_of_license_text": 91.75,
      "copyrights": [
        {
          "value": "Copyright (c) Isaac Z. Schlueter and Contributors",
          "start_line": 3,
          "end_line": 3
        },
        {
          "value": "Copyright Mathias Bynens <https://mathiasbynens.be/>",
          "start_line": 22,
          "end_line": 22
        }
      ],
      "holders": [
        {
          "value": "Isaac Z. Schlueter and Contributors",
          "start_line": 3,
          "end_line": 3
        },
        {
          "value": "Mathias Bynens",
          "start_line": 22,
          "end_line": 22
        }
      ],
      "authors": [],
      "packages": [],
      "emails": [],
      "urls": [
        {
          "url": "https://mathiasbynens.be/",
          "start_line": 22,
          "end_line": 22
        }
      ],
      "files_count": 0,
      "dirs_count": 0,
      "size_count": 0,
      "scan_errors": []
    }
  ]
}

We need to run the transform to map the about_resource field

field_renamings:
	about_resource : path

This is one of the ABOUT files

# Generated with AboutCode Toolkit Version 5.1.0 

about_resource: lic.txt
name: lic.txt
type: file
base_name: lic
extension: .txt
size: 2011
date: '2020-11-09'
sha1: fe4ff75272675c27279ddf101a2d6d7cd1e9ad64
md5: 326d5674181c4bb210e424772c60fa80
sha256: 3098be2717081cfe2b88a0faa221cb4f2326720cb8ce869199a9d6234f219319
mime_type: text/plain
file_type: ASCII text
is_text: yes
licenses:
  - key: isc
    score: '98.29'
    name: ISC License
    short_name: ISC License
    category: Permissive
    is_exception: no
    owner: ISC - Internet Systems Consortium
    homepage_url: https://www.isc.org/software/license
    text_url: http://fedoraproject.org/wiki/Licensing:MIT#Old_Style_with_legal_disclaimer_2
    reference_url: https://enterprise.dejacode.com/urn/urn:dje:license:isc
    spdx_license_key: ISC
    spdx_url: https://spdx.org/licenses/ISC
    start_line: 1
    end_line: 15
    matched_rule:
      identifier: isc_48.RULE
      license_expression: isc
      licenses:
        - isc
      is_license_text: yes
      is_license_notice: no
      is_license_reference: no
      is_license_tag: no
      matcher: 3-seq
      rule_length: 117
      matched_length: 115
      match_coverage: '98.29'
      rule_relevance: '100.0'
  - key: mit
    score: '100.0'
    name: MIT License
    short_name: MIT License
    category: Permissive
    is_exception: no
    owner: MIT
    homepage_url: http://opensource.org/licenses/mit-license.php
    text_url: http://opensource.org/licenses/mit-license.php
    reference_url: https://enterprise.dejacode.com/urn/urn:dje:license:mit
    spdx_license_key: MIT
    spdx_url: https://spdx.org/licenses/MIT
    start_line: 19
    end_line: 20
    matched_rule:
      identifier: mit_14.RULE
      license_expression: mit
      licenses:
        - mit
      is_license_text: no
      is_license_notice: no
      is_license_reference: yes
      is_license_tag: no
      matcher: 2-aho
      rule_length: 2
      matched_length: 2
      match_coverage: '100.0'
      rule_relevance: '100.0'
  - key: mit
    score: '100.0'
    name: MIT License
    short_name: MIT License
    category: Permissive
    is_exception: no
    owner: MIT
    homepage_url: http://opensource.org/licenses/mit-license.php
    text_url: http://opensource.org/licenses/mit-license.php
    reference_url: https://enterprise.dejacode.com/urn/urn:dje:license:mit
    spdx_license_key: MIT
    spdx_url: https://spdx.org/licenses/MIT
    start_line: 24
    end_line: 41
    matched_rule:
      identifier: mit.LICENSE
      license_expression: mit
      licenses:
        - mit
      is_license_text: yes
      is_license_notice: no
      is_license_reference: no
      is_license_tag: no
      matcher: 2-aho
      rule_length: 161
      matched_length: 161
      match_coverage: '100.0'
      rule_relevance: 100
license_expressions:
  - isc
  - mit
  - mit
percentage_of_license_text: '91.75'
copyrights:
  - value: Copyright (c) Isaac Z. Schlueter and Contributors
    start_line: 3
    end_line: 3
  - value: Copyright Mathias Bynens <https://mathiasbynens.be/>
    start_line: 22
    end_line: 22
holders:
  - value: Isaac Z. Schlueter and Contributors
    start_line: 3
    end_line: 3
  - value: Mathias Bynens
    start_line: 22
    end_line: 22
urls:
  - url: https://mathiasbynens.be/
    start_line: 22
    end_line: 22

Now the question is how do I present these data in CSV?
I was thinking to have something like licenses_key, licenses_score, copyrights_value, copyrights_start_line, holders_value etc.. i.e. we use the key as the prefix follow by an '_' then with the sub-name(key)
However, if we take this approach, when we want to use this CSV to generate ABOUT files, how do we identify which should be separated out to be the name of the list?

Suggestion/comments are welcome.

[mjherzog]

The primary purpose of this enhancement was to allow you to accept ScanCode data for Attribution without review or editing. In that context we need to go back to your comments from 12/9/2019. We need to filter out the start/line data for licenses, copyrights, emails or URLs and the matched_rule data for licenses.
If you are not going to analyze and conclude the ScanCode data then we should just pass it through, except that it would be nice to have an option to set the minimum license score for inclusion so that you would only accept license data greater than or equal to that score.

[chinyeungli]

well, 2 comments:

1. If the purpose is just to accept ScanCode data for Attribution without review or editing, then it wouldn't hurt to have those start/end line data and matched_rule data in the ABOUT files as shown above.
2. The current --fetch-license won't work as the AboutCode TK is looking for license_expression field which is a string field with "AND"/"OR" while the scancode use

      "license_expressions": [
        "isc",
        "mit",
        "mit"
      ],

Note that the field name is not the same (i.e. scancode uses plural) and the structure is different as well (i.e. scancode uses list instead of string)

I will keep digging and see what's the best solution..
I am thinking to have an option such as --from_scancode so that the tool know the input is from scancode and then can perform some pre-defined formatting... If no --from_scancode option is flagged, then all the non-supported fields will be kept and be considered as custom fields.