Copyright line numbers reported are incorrect for binary file

[rakeshbalusa]

OS: Windows 7 64-bit
Python version 2.7 (for windows 64 bit)
I am using the latest version of scancode.
When I scanned the file WebGrease.dll from the Nuget package Web Grease 1.6.0 (URL - https://www.nuget.org/api/v2/package/WebGrease/1.6.0)
scancode results are as follows :

Path                                What    Start Line  End Line        Info
C:/doc/temporary/WebGrease.dll  Copyright    8495            8501       Copyright Microsoft 2013 NWG, PublicKey

But when I opened the file WebGrease.dll using Notepad++ I found the text Copyright © Microsoft 2013 in line 31141

[pombredanne]

@rakeshbalusa thanks for this report. Beside the line numbers, the reported copyright is not entirely correct, but is there any copyright statement missing?

[rakeshbalusa]

@pombredanne There is another string CopyrightAttribute AssemblyTrademarkAttribute AssemblyCultureAttribute ComVisibleAttribute in line 28134... but it doesn't look like a copyright so, I didn't mention it

[pombredanne]

yes, this is correct and these are not copyrights

[sschuberth]

WebGrease.dll being a binary file, how much sense does it make to refer to line numbers in that case anyway? Maybe better report an offset and length (both in bytes) for Copyright findings in binary files?

[pombredanne]

@sschuberth These line number are returned for now for consistency but they are mostly meaningless as they highly depend on how strings were extracted from a binary (using the textcode.strings module) https://github.com/nexB/scancode-toolkit/blob/e45f8f401ca22a15fda6a78dff317e8acf94038e/src/textcode/strings.py
Because of the way strings are extracted, it is not possible to get character offsets short of some overhaul which I do not find palatable right away, since I plan to add support for wide characters sets strings (UTF-16-like) that are often found in Windows DLLs and also sometimes in Elfs.

Help would be mucho welcomed.
This is not entirely trivial because carrying around offsets would break a lot of client code for the strings module. An alternative is post-processing: once matches were found, re-parse the binary with some alternative but similar technique that would preserve offsets and then re-align the found matches back to this reparsing.

Note also even though lines are meaningless, they still have some little value for now: they are an indication of the ordering of matches when several matches are found in a file.

Another issue is that for now even for plain texts, the line numbers returned for copyrights are correct but not precise for copyrights. It is an indication of the lines range were the copyright was found, not the exact lines. This is a limitation of the underlying NLP parser from nltk for now (http://www.nltk.org/api/nltk.tag.html#nltk.tag.sequential.RegexpTagger) that does not track line number nor offsets.
To get exact line numbers for texts we could consider either submit patches to the upstream NLTK or look for alternative (such as https://github.com/spacy-io/spaCy). Or again apply a reparsing technique also for texts as suggested above.

Related but for licenses, and as part of #260, the actual matched license text should be optionally returned like we return the actual value for copyrights. Because there too, lines are meaningless for binaries matches. And with actual matched texts returned, line numbers matter less. They become just an indication of the relative ordering of a match.