Scancode generates incomplete copyright statement

[rbieniek]

Description

When scanning the github library https://github.com/thenativeweb/uuidv4.git on tag branch 1.0.0, the copyright statements should include the text: 'Copyright (c) 2014-2021 the native web' for the file 'README.md'.
The actual scan output is truncated to 'Copyright (c) 2014-2021 the'

The following versions of scancode were used to reproduce the issue:

• 3.2.1rc2
• 21.3.31
• 21.7.30

How To Reproduce

Checkout the uuidv4 github repository, switch to branch 1.0.0 and execute scancode.

Execute this sequence of commands:
$ git pull https://github.com/thenativeweb/uuidv4.git
$ (cd uuidv4; git checkout 1.0.0)
$ scancode --copyright --license --info --strip-root --timeout=300 $PWD/uuidv4 --json-pp $PWD/uuidv4-scan-result.yml

System configuration

The issue was reproduced on following OS versions:

• Ubunutu Bionic
• Ubunut Focal
• Fedore 34

The following scancode version were used for testing:

• 3.2.1rc2
• 21.3.31
• 21.7.30
  All scancode version were installed from the binary tarballs found on github

The projects README.md along with the output of scancode are attached to this ticket

uuidv4-README.md.txt
uuidv4-scan-result-v3.2.1rc2.yml.txt
uuidv4-scan-result-v21.3.31.yml.txt
uuidv4-scan-result-v21.7.30.yml.txt

uuidv4-README.md

[pombredanne]

@rbieniek @porsche-rishisaxena thank you for this thorough and detailed report

[pombredanne]

Note that "the native web" is not an easy case. Why? the copyright detection essentially uses a combo lexing and parsing to achieve a simple named entity recognition to find what is a name to use in a copyright statement. case, punctuation all matter there. The all lowercase "the native web" is also using non conspicuous and rather common words.

[nikpete]

@pombredanne My conclusion (based on your comments) would be that it cannot be solved due to the above mentioned reasons, right?

[philippsimon]

I also stumbled about that issue for uuidv4. Would be great if you could fix it.

Here btw their license text:
https://github.com/thenativeweb/uuidv4/blob/main/LICENSE.txt

The MIT License (MIT)
Copyright (c) 2014-2021 the native web.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

[pombredanne]

My conclusion (based on your comments) would be that it cannot be solved due to the above mentioned reasons, right?

@nikpete no, it can be solved .... this is not a quick fix like most other copyright detection issues

[pombredanne]

@philippsimon Thanks for the additional report!

[pombredanne]

For reference, copyright detection process is explained here:
https://github.com/nexB/scancode-toolkit/blob/5db9d05115bb6a3620f652bbbc9609ac6855bd48/src/cluecode/copyrights.py
The process consists in:

• prepare and cleanup text
• identify regions of text that may contain copyright (using hints).
  These are called "candidates".
• tag the text to recognize (e.g. lex) parts-of-speech (POS) tags to identify various copyright
  statements parts such as dates, companies, names ("named entities"), etc.
  This is done using pygmars which contains a lexer derived from NLTK POS tagger.
• feed the tagged text to a parsing grammar describing actual copyright
  statements (also using pygmars) and obtain a parse tree.
• Walk the parse tree and yield copyright statements, holder and authors with start
  and end line from the parse tree with some extra post-detection cleanups.

The lexing step uses a single pass using a single token at a time here https://github.com/nexB/pygmars/blob/bf26647555c3771b8a0c36c7685abfc18267957c/src/pygmars/lex.py#L131 using these lexing patterns: https://github.com/nexB/scancode-toolkit/blob/5db9d05115bb6a3620f652bbbc9609ac6855bd48/src/cluecode/copyrights.py#L514
Then the grammar at https://github.com/nexB/scancode-toolkit/blob/5db9d05115bb6a3620f652bbbc9609ac6855bd48/src/cluecode/copyrights.py#L1673 is used to parse the tokens stream and detect all the various forms of author and copyright statements.

With these, "the" "native" "web" will all be detected as plain NN aka. plain Nouns. Plain NN nouns are not featured prominently in the grammar to detect NAME or COMPANY constructions because they seldom are part of these, and they surely are rarely (except here) composed entirely of NNs; Most NAMEs are composed of proper nouns with some possible suffix/prefix like A.G. or Inc. "the" "native" "web" could be detected instead as NNP aka. Proper Nouns but this would create a lot of false positive detections as these are common words which are not Proper Nouns at all and would skew the way the
grammar detects things.

Instead we could either:

• add a way to do second pass on the lexed tokens stream

• m that considers multiple tokens at a time to re-assign each a token type; here "the" "native" "web" tokens could be re-assigned each with an NNP token label when seen in this sequence. This stream of "the"/NNP "native"/NNP "web"/NNP tokens will then be parsed further by the existing grammar Rules in a NAME of a COPYRIGHT statement

• add an alternative grammar construct such that grammar Rules can take into account not only the Token type but also the Token value. Today the Rules are patterns defined over the lexed tokens Labels https://github.com/nexB/pygmars/blob/bf26647555c3771b8a0c36c7685abfc18267957c/src/pygmars/parse.py#L448 . With this a token sequence of a combo of value/label such as "the"/NN "native"/NN "web"/NN tokens could be then be parse as a NAME which will then be parsed further by the existing grammar Rules in a COPYRIGHT statement

Either approach would take about two days worth of work to implement. Do you think someone could help there? 😇

[pombredanne]

BTW to highlight the solution further, with this text in a file named c:

• Copyright (c) 2014-2021 The Native Web.
  and when I run a copyright scan:
• SCANCODE_DEBUG_COPYRIGHT=yes scancode --copyright --json-pp - c
  ... this is detected correctly as "Copyright (c) 2014-2021 The Native Web"
  and the key debug log lines are:

DEBUG:cluecode.copyrights:CopyrightDetector: initial tokens: [(label=None, value='Copyright'), (label=None, value='(c)'), (label=None, value='2014-2021'), (label=None, value='The'), (label=None, value='Native'), (label=None, value='Web.')]

DEBUG:cluecode.copyrights:CopyrightDetector: lexed tokens: [(label='COPY', value='Copyright'), (label='COPY', value='(c)'), (label='YR', value='2014-2021'), (label='NN', value='The'), (label='NNP', value='Native'), (label='NN', value='Web.')]

DEBUG:cluecode.copyrights:CopyrightDetector: parse_tree: (label='ROOT', children=(
    (label='COPYRIGHT2', children=(
    (label='COPY', value='Copyright')
    (label='COPY', value='(c)')
    (YR-RANGE (YR-RANGE (label='YR', value='2014-2021')))
    (label='NN', value='The')
    (label='NNP', value='Native')
    (label='NN', value='Web.')
  ))
))
DEBUG:cluecode.copyrights:CopyrightDetector: detection: Detection(value='Copyright (c) 2014-2021 The Native Web', type='copyrights', start_line=1, end_line=1)
DEBUG:cluecode.copyrights:CopyrightDetector: holders: Detection(value='The Native Web', type='holders', start_line=1, end_line=1)
DEBUG:cluecode.copyrights: detect_copyrights_from_lines: [Detection(value='Copyright (c) 2014-2021 The Native Web', type='copyrights', start_line=1, end_line=1), Detection(value='The Native Web', type='holders', start_line=1, end_line=1)]

And the key grammar logging is with SCANCODE_DEBUG_COPYRIGHT_DEEP=yes scancode --copyright --json-pp - c

# Input parsed as label: 'YR-RANGE'
  before: ' <COPY>  <COPY>  <YR>  <NN>  <NNP>  <NN>'
  after : ' <COPY>  <COPY> {<YR>} <NN>  <NNP>  <NN>'

# Input parsed as label: 'YR-RANGE'
  before: ' <COPY>  <COPY>  <YR-RANGE>  <NN>  <NNP>  <NN>'
  after : ' <COPY>  <COPY> {<YR-RANGE>} <NN>  <NNP>  <NN>'

# Input parsed as label: 'COPYRIGHT2'
  before: ' <COPY>  <COPY>  <YR-RANGE>  <NN>  <NNP>  <NN>'
  after : '{<COPY>  <COPY>  <YR-RANGE>  <NN>  <NNP>  <NN>}'

In contrast when using the all lower case:
we get:

# Input parsed as label: 'YR-RANGE'
  before: ' <COPY>  <COPY>  <YR>  <NN>  <NN>  <NN>'
  after : ' <COPY>  <COPY> {<YR>} <NN>  <NN>  <NN>'

# Input parsed as label: 'YR-RANGE'
  before: ' <COPY>  <COPY>  <YR-RANGE>  <NN>  <NN>  <NN>'
  after : ' <COPY>  <COPY> {<YR-RANGE>} <NN>  <NN>  <NN>'

# Input parsed as label: 'COPYRIGHT'
  before: ' <COPY>  <COPY>  <YR-RANGE>  <NN>  <NN>  <NN>'
  after : '{<COPY>  <COPY>  <YR-RANGE>  <NN>} <NN>  <NN>'

the last rule used is this one:
https://github.com/nexB/scancode-toolkit/blob/5db9d05115bb6a3620f652bbbc9609ac6855bd48/src/cluecode/copyrights.py#L2218

Changing it from:

• COPYRIGHT: {<COPY>+ <YR-RANGE> <NN> <ALLRIGHTRESERVED>?} #22793
  to
• COPYRIGHT: {<COPY>+ <YR-RANGE> <NN>+ <ALLRIGHTRESERVED>?} #22793

make the detection works on the all lowercase variants... but then running the tests yields these detection failures:

E             - data file: file:///home/scancode-toolkit-master/tests/cluecode/data/ics/oprofile-utils/opcontrol.yml
E             - test file: file:///home/scancode-toolkit-master/tests/cluecode/data/ics/oprofile-utils/opcontrol
E               what:
E                 - copyrights
E                 - holders
E                 - holders_summary
E               copyrights:
E             -   - Copyright 2002 Read
E             +   - Copyright 2002 Read the
E             ?                        ++++
E                 - Copyright IBM Corporation 2007
E               holders:
E             -   - Read
E             +   - Read the
E             ?         ++++
E                 - IBM Corporation
E               holders_summary:
E                 - value: IBM Corporation
E                   count: 1
E             -   - value: Read
E             +   - value: Read the
E             ?                ++++
E                   count: 1

tests/cluecode/cluecode_test_utils.py:219: AssertionError


E             - data file: file:///home/scancode-toolkit-master/tests/cluecode/data/ics/libvpx-examples-includes-geshi-docs/geshi-doc.html.yml
E             - test file: file:///home/scancode-toolkit-master/tests/cluecode/data/ics/libvpx-examples-includes-geshi-docs/geshi-doc.html
E               what:
E                 - copyrights
E                 - holders
E                 - holders_summary
E               copyrights:
E                 - (c) 2004 - 2007 Nigel McNie, 2007 - 2008 Benny Baumann (http://qbnz.com/highlighter/ and
E                   http://geshi.org/)
E                 - (c) 2004 - 2007 Nigel McNie, 2007 - 2008 Benny Baumann (http://qbnz.com/highlighter/ and
E                   http://geshi.org/)
E                 - (c) 2004 - 2007 Nigel McNie, 2007 - 2008 Benny Baumann (http://qbnz.com/highlighter/ and
E                   http://geshi.org/)
E                 - (c) 2004 - 2007 Nigel McNie, 2007 - 2008 Benny Baumann (http://qbnz.com/highlighter/ and
E                   http://geshi.org/)
E                 - (c) 2004 - 2007 Nigel McNie, 2007 - 2008 Benny Baumann (http://qbnz.com/highlighter/ and
E                   http://geshi.org/)
E                 - Copyright (c) 2004 Nigel McNie (http://qbnz.com/highlighter/)
E             -   - Copyright (c) 2008 name
E             +   - Copyright (c) 2008 name ( website URL )
E             ?                            ++++++++++++++++
E               holders:
E                 - Nigel McNie, - Benny Baumann
E                 - Nigel McNie, - Benny Baumann
E                 - Nigel McNie, - Benny Baumann
E                 - Nigel McNie, - Benny Baumann
E                 - Nigel McNie, - Benny Baumann
E                 - Nigel McNie
E             -   - name
E             +   - name  website URL
E               holders_summary:
E                 - value: Nigel McNie, - Benny Baumann
E                   count: 5
E                 - value: Nigel McNie
E                   count: 1
E             -   - value: name
E             +   - value: name website URL
E                   count: 1

tests/cluecode/cluecode_test_utils.py:219: AssertionError

oh well.... these are only two failures! and they may be improvements in fact in these case.
So there may be a quick fix after all :)

[philippsimon]

@pombredanne thanks a lot!!