mass-scan: Design and create code to spot incorrect copyrights

[pombredanne]

I would like to spot incorrect copyright detections, the ones that would need a human review afterwards.

• Design and create code to spot incorrect copyrights in ScanCode.io and PurlDB
• Apply AI/ML to spot copyrights issues and generate automated fixes/improvement

Some references:

• Improve gibberish license/copyright detection scancode-toolkit#2402

[JonoYang]

aboutcode-org/scancode-toolkit#2402 (comment)

We now have code that performs gibberish detection on strings using a 2-character markov chain, which has been merged in aboutcode-org/scancode-toolkit#4610

At a high level, it detects gibberish strings by seeing if a sequence of letters is part or a whole word, two letters at a time. It does this by checking how likely it is to go from one letter to another. The probabilities of going from one letter to another are determined by a model that has been trained on a large set of valid text, where it counts each transition between letters and computes a probability based off of that. These probabilities and thresholds are stored in a model that is saved to a Python pickle.

These copyrights are no longer reported.

These fixes are now in scancode-toolkit and will be available in future releases. These enhancements will also be available in scancode.io .