The --license-text does not work as expected (for JSON output)

[sschuberth]

I would expect the --license-text option to put the full license text as defined by the start and end lines into the JSON result file. However, it seems this option only adds a key called matched_text which only contains the excerpt of the license text that triggered the match. See e.g. this result from scanning re2's BUILD file:

  "licenses": [
    {
      "key": "bsd-new",
      "score": 92.31,
      "short_name": "BSD-Modified",
      "category": "Permissive",
      "owner": "Regents of the University of California",
      "homepage_url": "http://www.opensource.org/licenses/BSD-3-Clause",
      "text_url": "",
      "dejacode_url": "https://enterprise.dejacode.com/urn/urn:dje:license:bsd-new",
      "spdx_license_key": "BSD-3-Clause",
      "spdx_url": "http://spdx.org/licenses/BSD-3-Clause",
      "start_line": 1,
      "end_line": 3,
      "matched_rule": {
        "identifier": "bsd-new_25.RULE",
        "license_choice": false,
        "licenses": [
          "bsd-new"
        ],
        "matcher": "3-seq",
        "rule_length": 26,
        "matched_length": 24,
        "match_coverage": 92.31,
        "rule_relevance": 100
      },
      "matched_text": "Authors.  All Rights Reserved.\n# Use of this source code is governed by a BSD-style\n# license that can be found in the LICENSE file"
    }

In this example, I would have expected matched_text to contain "# Copyright 2009 The RE2 Authors. All Rights Reserved.\n# Use of this source code is governed by a BSD-style\n# license that can be found in the LICENSE file." (also note the trailing dot at the end of the sentence here).

Also, matched_text seems to be defined at the wrong level. As it's value should be defined by the contents of the file from start_line to end_line, it should also be defined at the level of start_line / end_line.

If changing the semantics of matched_text is not possibe for compatibility reasons, I'd suggest to create a new key called license_text listed directly after end_line containing the specified lines of the scanned file.

[pombredanne]

@sschuberth Thanks. Let me review this. The matched text is still experimental and indeed only returns the matched text. And to your point it would help to expand it.
Two remarks:

• bsd-new_25.RULE should unlikely contain this text Authors. All Rights Reserved. This is a part of copyright statement and not the license text proper
• I think the full text should be some deduction based on the matched text(s) (and the unmatched parts in between matches), the detected copyrights, emails, urls, and how much of these are covering the underlying text and a smart reassembly of all these in a text eventually as a deduction-like process (though since this is a a more discrete process, this does not need all the sophistication planned for Proposal: Scan deduction and summarization #377

[sschuberth]

Let me give you some background information on our intended use of --license-text: Basically, we'd like to automatically create NOTICE files by concatenating the contents of the values of the respective JSON keys. As such we're not interested in the details of which text was matched, but really the full license text as it appears in the files. Of course we could manually read these from the files using the start_line and end_line values, but we do not always have access to the scanned files anymore, only to the JSON result, thus it would be good to have these lines in the result.

[pombredanne]

@sschuberth I agree with the goal: create NOTICE files automatically. Let me dig how this can be done

[pombredanne]

so I got some informal request for this today too, so I will try to give it a shot
Note that grabbing whole lines at first should work OK, but may break when:

• this is a binary
• this is text with very long lines.

So I will try yo get something decent for these cases.

[haikoschol]

@pombredanne you mentioned the matched_text field is experimental. I wonder what the use case for it is and if it would still be served by including the full text.

I'm looking into this ticket now and would like to know what you prefer: a new command line option and attribute in the JSON result, or changing matched_text to include the full text. In the latter case it might still make sense to rename it, though. In it's current form matched_text looks more like a debugging aid for working on ScanCode itself.

[pombredanne]

We want matched text come up with --license-text eventually with whole lines when relevant and optional highlighting of unmatched parts (and not be default as this is only useful for debugging or deep curation)

A goal is to come with something that could eventually make a scan be usable for attribution pretty much as-is and fed to https://github.com/nexB/aboutcode-toolkit for formatting a nice attribution doc. It may not be perfect, but pretty darn good!

[pombredanne]

I am fine to make this a high level attribute instead of the debug one