Improve gibberish license/copyright detection

[JonoYang]

I have compiled a text file that contains erroneous copyright detection values. I have removed quote characters and separated each copyright value by several lines.
bad-copyright-detections.txt

[pombredanne]

That's an interesting class of errors!
I guess that they all come from binaries? And because it is useful, we cannot stop detecting in binaries.

Some remarks:
These are at first considered as candidates by the copyright detector because they contains (c) or © (which is normalized to (c)). Then they are somehow matching some of the lexing rules in our part of speech tagger and are tagged as being recognized names somehow, and as names they end up recognized finally by the grammar.

1. We could refine the candidate detection when in binaries such that we are more strict in that case?
2. We could improve the crude/dump "Named Entity Recognition"(NER) we have today to skip some. There are features in NLTK and we could give a shot to https://github.com/explosion/spaCy and several other NLP tools that help there
3. We could detect gibberish, either early in candidate or NER/tagging or after the fact in a post scan plugin. For that https://github.com/casics/nostril/ which is now LGPL (and no longer LGPL) Would you consider a license suitable for use as a library in non GPL apps? casics/nostril#9
   That's a gibberish detector.

[pombredanne]

Note that the adoption of https://github.com/nexB/pygmars/ as a replacement for NLTK should allow the easier reuse and integration of other libraries in the lexing process including NER and giberish detection. In pygmars, a tokenization rule can now be an arbitrary callable that behaves like re.match so it may enable using other tools as part of token recognition aka. lexing

[pombredanne]

Another candidate for gibberish that works quite well is https://github.com/domanchi/gibberish-detector

[pombredanne]

See

• C source code line mistaken for MPL licence  #2304
• Discard matches to single GPL word and other very short rules with mixed, non-matching case and/or in a binary an/or not on a single line and/or in giberish  #2403

[pombredanne]

I ran this with bad-copyright-detections.txt

• pip install gibberish-detector
• gibberish-detector train examples/big.txt > big.model
• in python:

from gibberish_detector import detector
Detector = detector.create_from_model('big.model')
data = sorted(set(open('bad-copyright-detections.txt').read().split()))
for d in data:
    print(repr(d),',', Detector.is_gibberish(d))

I then loaded this in libreoffice to do some evaluation. Here are the results:

string	gibberish	correct
!e	False	no
?a	False	no
.	False	no
(c)	False	no
(c)i	False	no
(c)u	False	no
\x00	False	no
\x0e	False	no
\x191/2/\x18$?R	False	no
A	False	no
A-	False	no
A\x7f\x7fl?	False	no
A&	False	no
A1	False	no
A8l	False	no
B	False	no
C	False	no
Cakd	False	no
Ci	False	no
co	False	yes
co9	False	no
Copyright	False	yes
CZal	False	no
D	False	no
E	False	no
E?	False	no
E1	False	no
E2	False	no
Eu	False	no
Ev	False	no
Evo	False	no
ew	False	no
FBee	False	no
G	False	no
H	False	no
H1o	False	no
I	False	no
io-ha\x15deg\x12q?	False	no
Mr	False	no
N	False	no
N1/2	False	no
NaluAd	False	yes
O	False	no
OEm	False	no
Ok\x19s1	False	no
Optional	False	yes
RO	False	yes
Sthg	False	yes
Thw	False	yes
U	False	no
Wad?	False	no
WrapNonExceptionThrows	False	yes
Xr66	False	no
Y	False	no
YO	False	no
(c)s	True	yes
\x00\x03\x00$LN2007\x00A\x00\x00\x00\x00\x00\x00\x06\x00$LN2008\x00th\x00\x00\x00\x00\x00\x00\x06\x00$LN2005\x00\x14	True	yes
\x7fCdegA-Aa*	True	yes
$?OAE+-OA	True	yes
1NE3	True	yes
1PS1	True	yes
3/4R\x08U*	True	yes
3\x12ae.Oi6	True	yes
A2C/a2EPSOA-D1/4	,C/3A True	yes
Aa	True	yes
AaASSA	True	yes
AAE	True	no
AaU	True	yes
AC3E	True	yes
AE1rvfZaUNXNJXNJ	True	yes
aeaaeO1/2	True	yes
AEIQDfR	True	yes
AEo	True	yes
AErA	True	yes
Ajc	True	yes
aMp@KPj\x13o*O BNeAa!'o-O\x06\x11R\x1boX\x04	True	yes
aO1rqaetet\x19il	True	yes
Ao3Vs	True	yes
AoD$?xAAA	True	yes
AoDIA	True	yes
ASS	True	yes
ASS?SSE-E2I*	True	yes
AThY	True	yes
AuNTO	True	yes
BjAO	True	yes
bvnay$?\x08xVx,i\x0fveoPSS	True	yes
C/ae3e,!cU	True	yes
C2CDCgC	True	yes
Cd\x03AU,AEoAd\x03AU,AEoA$?xAAA	True	yes
Cdegs	True	yes
CkI3H	True	yes
CR	True	yes
Cu	True	yes
CuN	True	yes
D8DYDoD	True	yes
Dlss	True	yes
dnOWEWly1	True	yes
DoPSOA	True	yes
Dq	True	yes
E,C/A+-I1PSxA(r)OA!O3/4OAaOAaO3/4	True	yes
E2EuA	True	yes
E3P	True	yes
EdegA	True	yes
Ee	True	yes
EP	True	yes
EuI1Y	True	yes
EvoS	True	yes
FaUSS1u	True	yes
FKo2	True	yes
FPS	True	yes
G3eeAIUq5w9\x11Wis2Wu\x000PS	Gae2	True
HPS3Ho	True	yes
I1PSEu	True	yes
I1PSIoY	True	yes
I1Y	True	yes
I3A	True	yes
IA5e	True	yes
IC-cC_	True	yes
IeIc	True	yes
IIIdOi09QBn	True	yes
IjKMr	True	yes
IldY	True	yes
Imu7UTa7eeem	True	yes
IOuOeO	True	yes
IoY	True	yes
IP	True	yes
Iu	True	yes
IU1u	True	yes
IUU	True	yes
IvR33	True	yes
IYte9thdegA8Ioie5	True	yes
Jw6	True	yes
Ks	True	yes
n$i1/44ThYV\x10th\x07C.'cOnVAoE	True	yes
nA+	True	yes
NE82	True	yes
NicI	True	yes
NN	True	yes
Nxe	True	yes
o//O\x7f\x17\x12O	True	yes
OA	True	yes
OAaE3E33/4	True	yes
OAaOA	True	yes
OaC2A	True	yes
OACn	True	yes
OAdegOA!ThEPaN	True	yes
Oae	True	yes
OAEauy	True	yes
OC2YI1UC2OA	True	yes
OE	True	yes
OeA-?Fbt3\x19Uea	True	yes
OeYeAE	True	yes
OPSE3e	True	yes
OuA	True	yes
Ox	True	yes
OXMssU	True	yes
OyA	True	yes
OYEYG4w2hssaeNUhbbAJMi	True	yes
OyN	True	yes
OYThx	True	yes
PI5uWi(r)&O\x08unO	True	yes
PJ	True	yes
PSa	True	yes
QAPYINae2A	True	yes
QLdXPo	True	yes
RSDS	True	yes
SA	True	yes
Sl	True	yes
SP	True	yes
SS	True	yes
SSA	True	yes
TCYu	True	yes
ThA3I\x18	True	yes
ThIoO(r)D1/2	True	yes
ThrEIuexOUoth	True	yes
Ua	True	yes
UAA7a	True	yes
UAA7ao	True	yes
UE	True	yes
UE3	True	yes
UE9HSS	True	yes
UIuUGw7	True	yes
UO	True	yes
Uos3	True	yes
V1oVaoA	True	yes
VDOI	True	yes
VkSS	True	yes
WY	True	yes
xA	True	yes
Xi	True	yes
XY	True	yes
yA!uAW	True	yes
YC	True	yes
YUUuIUI	True	yes
YV	True	yes

This is pretty good and I would expect even better from using a proper training set.

[AyanSinhaMahapatra]

I was testing out the very basic gibberish detector at https://github.com/rrenaud/Gibberish-Detector, which https://github.com/domanchi/gibberish-detector (mentioned above is based on), as it is much easier to integrate.

I used our scancode license texts and rules (minus the test set) as the training data and false positives and some of the license tags as test data.

It is pretty good at detecting non gibberish but is not so good when these are ambigious.

attaching the results of the test here for reference, here the probability is of the text being non-gibberish.

bad -> false positives
good -> license tags

gibberish-test.csv

[pombredanne]

very nice! what's your take on applicability to license then? Did you apply some boosting to legalese words?

[AyanSinhaMahapatra]

I still think we need better performance to integrate, i was looking into the other implementation which is a library. There are some additional steps there so I'll try that with the same data. Also thing to note is this only uses positive training, i.e. only trains on good non gibberish values, so if we could do some negative boosting for legalese gibberish that could improve the performance.

I did extend the character set to include a-z, A-Z, number and other characters.

Do you think it's worth spending time on this?

[pombredanne]

Do you think it's worth spending time on this?

Let's be mindful not to get too much into the weeds as this can be hairy and yield only small improvements.
It may be best to start using this on copyright before applying to license?

[pombredanne]

Some samples for copyrights:

• https://files.pythonhosted.org/packages/ee/2d/9cdc2b527e127b4c9db64b86647d567985940ac3698eeabc7ffaccb4ea61/chardet-4.0.0.tar.gz
• https://github.com/microsoft/Azure-Social-Media-Analytics-Solution-Accelerator/blob/bd10c3af5779943bd31bd859ee894a2960cded8b/NOTICE.txt#L254

[pombredanne]

Here is the list of gibberish originally attached:

(c) !e (c) YC UO
(c) (c) RO
(c) . (c)
(c) . Optional
(c) . SS
(c) . SSA
(c) 1NE3 AuNTO
(c) 3/4R\bU* OYEYG4w2hssaeNUhbbAJMi n$i1/44ThYV:\u0010th\u0007C.'cOnVAoE aO1rqaetet\u0019il io-ha\u0015deg\u0012q? ASS
(c) ?a Wad? E ASS
(c) \x00 \x00\x03\x00$LN2007\x00A\x00\x00\x00\x00\x00\x00\x06\x00$LN2008\x00th\x00\x00\x00\x00\x00\x00\x06\x00$LN2005\x00\x14
(c) \x7fCdegA-Aa* (c) A
(c) A OXMssU
(c) A Sthg
(c) A- yA!uAW (c) A
(c) A1 (c)
(c) A2*C/a2E*PSOA-D1/4|,C/3A (c) E
(c) A8l OyA
(c) A\x7f\x7fl? (c) ASS?SSE-E2I*
(c) Aa (c)i
(c) AaASSA (c)
(c) AAE IA5e
(c) AC3E Eu
(c) AE1rvfZaUNXNJXNJ QLdXPo
(c) aeaaeO1/2 (c) E3P
(c) AEo (c)
(c) aMp@KPj\u0013o*O:BNeAa!'o-O\u0006\u0011R\u001boX\u0004 (c) N
(c) Ao3Vs NicI
(c) AoD$?xAAA Cd\x03AU,AEoAd\x03AU,AEoA$?xAAA C nA+ A SA
(c) AoDIA Y
(c) AThY (c)
(c) C2CDCgC D8DYDoD B
(c) Cakd (c)
(c) Cdegs NE82
(c) co
(c) CZal (c)
(c) D AEIQDfR
(c) D Evo
(c) D EvoS
(c) Dlss BjAO OYThx
(c) DoPSOA (c)
(c) E1 (c)
(c) E2EuA (c)
(c) E? (c) 1PS1
(c) EdegA (c)
(c) Ee VDOI
(c) EP EuI1Y
(c) ew (c) OE
(c) FKo2 (c)
(c) FPS FBee
(c) G HPS3Ho
(c) G3eeAIUq5w9\u0011Wis2Wu\u00000PS|Gae2 (c) IUU CR
(c) H OPSE3e
(c) H1o I
(c) I $?OAE+-OA (c) I
(c) I Nxe UE9HSS
(c) I1PSEu Eu
(c) I1PSIoY Eu
(c) IC-cC_ Ci
(c) IIIdOi09QBn C/ae3e,!cU UAA7a
(c) IIIdOi09QBn C/ae3e,!cU UAA7ao
(c) IjKMr OEm
(c) Imu7UTa7eeem (c)
(c) IoY E2
(c) IoY IoY
(c) IP Iu
(c) IYte9thdegA8Ioie5 (c)
(c) Jw6 E
(c) Mr Ajc
(c) N1/2 (c) I1Y
(c) NaluAd (c)
(c) o//O\x7f\x17\x12O xA OeA-?Fbt3\x19Uea \x191/2/\x18$?R co9
(c) OAaOA (c)
(c) OaC2A (c)
(c) OACn (c)
(c) OAdegOA!ThEPaN OAaE3E33/4 (c) Y
(c) OC2YI1UC2OA E,C/A+-I1PSxA(r)OA!O3/4OAaOAaO3/4 (c) I3A
(c) OeYeAE A& IldY
(c) Ok\x19s1 (c) O
(c) Ok\x19s1 (c) OA
(c) Ox (c)s
(c) Ox (c)u
(c) OyN Y
(c) PI5uWi(r)&O\x08unO 3\x12ae.Oi6 FaUSS1u bvnay$?\x08xVx,i\x0fveoPSS
(c) PJ Dq
(c) QAPYINae2A (c)
(c) Sl SP
(c) TCYu U
(c) ThA3I\u0018 (c) U
(c) Thw NN
(c) Uos3 YO
(c) Uos3 YV
(c) V1oVaoA (c)
(c) WY Cu
(c) WY CuN
(c) Xi ThrEIuexOUoth
(c) Xr66 (c)
(c) XY IU1u UE
(c) XY IU1u UE3
AErA \u000e| (c) Ua
Copyright WrapNonExceptionThrows RSDS
Eu ThIoO(r)D1/2 (c) IoY
Ev (c) CkI3H
IOuOeO (c) Oae
OuA (c) OAEauy
PSa (c) IvR33
UIuUGw7 (c) AaU
VkSS dnOWEWly1 (c) Ks
YUUuIUI (c) IeIc