Does not detect all author names in Cargo.toml

[ritiek]

Description

Hi, I ran scancode to detect copyright information but it does not correctly detect authors from Cargo.toml file (which get used in Rust crates). For example:

$ wget https://raw.githubusercontent.com/rust-lang/rust-clippy/master/Cargo.toml
$ scancode Cargo.toml --copyright --json-pp cargo.json
$ cat cargo.json
{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "3.0.2.post331.45bbafd17",
      "options": {
        "input": "Cargo.toml",
        "--copyright": true,
        "--json-pp": "cargo.json"
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2019-03-07T064943.463630",
      "end_timestamp": "2019-03-07T064943.891931",
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "Cargo.toml",
      "type": "file",
      "holders": [],
      "copyrights": [],
      "authors": [
        {
          "value": "Manish Goregaokar <manishsmail@gmail.com> , Andre Bogus <bogusandre@gmail.com>",
          "start_line": 4,
          "end_line": 6
        }
      ],
      "scan_errors": []
    }
  ]
}

Here it displays only 2 authors:

"Manish Goregaokar <manishsmail@gmail.com> , Andre Bogus <bogusandre@gmail.com>"

where as there are 5 authors mentioned in the acutal Cargo.toml:

	"Manish Goregaokar <manishsmail@gmail.com>",
	"Andre Bogus <bogusandre@gmail.com>",
	"Georg Brandl <georg@python.org>",
	"Martin Carton <cartonmartin@gmail.com>",
	"Oliver Schneider <clippy-iethah7aipeen8neex1a@oli-obk.de>"

System configuration

• What OS are you running on?
  Linux
• What version of scancode-toolkit was used to generate the scan file?

$ scancode --version
ScanCode version 3.0.2.post331.45bbafd17

• What installation method was used to install/run scancode?
  source (45bbafd)

---

If you guys have any ideas, I'd be glad to fix this myself. :)

[ritiek]

I noticed that if all the authors are mentioned in a single line (no line breaks), scancode works fine. That is if I replace

authors = [
	"Manish Goregaokar <manishsmail@gmail.com>",
	"Andre Bogus <bogusandre@gmail.com>",
	"Georg Brandl <georg@python.org>",
	"Martin Carton <cartonmartin@gmail.com>",
	"Oliver Schneider <clippy-iethah7aipeen8neex1a@oli-obk.de>"
]

with

authors = ["Manish Goregaokar <manishsmail@gmail.com>", "Andre Bogus <bogusandre@gmail.com>", "Georg Brandl <georg@python.org>", "Martin Carton <cartonmartin@gmail.com>", "Oliver Schneider <clippy-iethah7aipeen8neex1a@oli-obk.de>"]

in the passed Cargo.toml, scancode is correctly able to detect all authors and their respective email.

It would be a good idea to add support for such line breaks in a candidate line.

[pombredanne]

@ritiek good catch! actually we would need to properly parse a cargo file the same way it is done for npm package.json, maven pom.xml and so on... that was a question that @fossygirl asked a while back.
So rather than dealing with the specifics (or shortcoming) of author detection in the cluecode module, we should add a proper cargo parser and normalizer in the packagedcode module, similar to any other manifest parsers in https://github.com/nexB/scancode-toolkit/tree/develop/src/packagedcode ....

[pombredanne]

@ritiek is this something you feel like having a look at and contribute? that would be awesome!

[ritiek]

@pombredanne Sure, I'll take a look.

[pombredanne]

the key is that we have a common, normalized data model for all package types and we feed this model by parsing the manifest. Each package type is a subclass of the base model such as a simple https://github.com/nexB/scancode-toolkit/blob/887ae131c3809a6a002267eebf8f2d5e817499f9/src/packagedcode/models.py#L679 or a more fleshed out thing for npms.

So here the path would likely be:

1. collect some representative Cargo files to use as a test set
2. create a new cargo.py module with a simple subclass of Package
3. add toml as a dep (see how this is done in thirdparty/ and setup.py)
4. create a mapping function that converts and normalizes cargo data into a CargoPackage ... get some inspiration from existing manifest parsers

(And later on, we can refine things for the detection, a suite of more data-driven tests, dealing with lock files, etc)