Copyright is incomplete when there is a newline between

[MMarwedel]

Hi,
if there is a newline between the copyright and the name of the company, the company does not show up in the "copyrights": as value.
Sample which goes wrong (actually not exactly the file I scanned, but same formatting and from the same company):
https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/include/newlib/sys/features.h
"copyrights": [
{
"value": "COPYRIGHT (c) 1989-2014.",
"start_line": 4,
"end_line": 4
}
],

Sample which works properly:
https://github.com/esp8266/Arduino/blob/master/tools/sdk/libc/xtensa-lx106-elf/include/sched.h
"copyrights": [
{
"value": "COPYRIGHT (c) 1989-2010. On-Line Applications Research Corporation",
"start_line": 4,
"end_line": 5
}
],

[pombredanne]

@MMarwedel Thank you for this. New, empty lines are used to decide which lines should be considered for a copyright detection. And empty line is this context is considered as a break and not part of the detection. The difficulty is to determine when this new lines is relevant and a break and when this is not not. In the vast majority of the cases, this is a break.
Is this something you have seen often or just a few odd times?

[MMarwedel]

I agree, its difficult to decide which part of the text belongs to which other part. In fact I only found one file with some "(c) DATE" but no autor or company following in the results.
I however cant say if I missed some
"(c) DATE author1 and

author2"

cases.

[pombredanne]

as weird as it sounds the likely best way to fix this is to fix this in newlib with a patch that would remove the empty line. I ran a quick test with a naive way to consider an empty as possibly part of a copyright statement and it created massive test failures. This is really uncommon.
I sent a patch: espressif/arduino-esp32#2787

As for cases "(c) DATE author1 and

author2"
they would be really weird and would be even harder to parse. It really looks like the established convention is not to put empty lines in the middle of a copyright statements. So if there is one, we have to assume that this is either a mistake and we can help fix this or this is not part of such statement.

[MMarwedel]

Okay, patching this in newlib, I guess is the best solution. The right place might be https://www.sourceware.org/newlib/ but I failed to find the file there and provide a link, so espressif was just a random other place to find the file online.

[pombredanne]

so it would be great if you know where your copy of newlib comes from exactly? there are many processor-specific variants that float around!

[MMarwedel]

Hi,
the file is from the source package provided by ARM:
https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads#panel2a
The "7-2018-q2-update" then the "Source Invariant" download.
Then in the gcc-arm-none-eabi-7-2018-q2-update\src
extract newlib.tar.bz2
then the file will be under
newlib/newlib/libc/include/sys/features.h

As ARM states on their site:
https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm
they get the source from
https://sourceware.org/newlib/
but I fail to use the online browsing to find the file and dont want to check out the whole git.

[pombredanne]

I am adding a few heuristics in candidate detection such that an empty line following a line ending with a year or AND or BY is not treated as a break in the stream of candidates. This will handle your cases at least and a few more

[pombredanne]

So upstream removed the code in question entirely and never merged espressif/esp-idf@9a2af7a ... 🙄 which is a solution for the future

But the many recent improvements to copyright detection are also fixing this. The results are now:

copyrights:
    -   value: COPYRIGHT (c) 1989-2000. On-Line Applications Research Corporation
        start_line: 1
        end_line: 3
holders:
    -   value: On-Line Applications Research Corporation
        start_line: 3
        end_line: 3

I think we are set now. Thanks!