I have mixed feelings on data files: I want to scan them as they may contain licenses or clues AND I do not want to scan them if they are highly likely not to contain any clues.
For the copyright detection that seeks possible dates and date ranges (such as Copyright (c) 2000-2013 XXXX), long list of numbers are a hog for now.
There are other cases where scanning data files is not great especially when these are big files.
Or for instance the firmwares are mostly hex blobs in the kernel. There I still want to scan them even if these may be huge hex blob-like: they have the most byzantine licenses of all and are worth reporting.
So we need a better way to skip scanning certain files that are pure data.
And in these cases just issue a warning that they were not scanned fully and likely only scan the first and last hundred lines.
There are a couple patterns like binary-only data files which are easy to ignore.
Or text files with lines made of digits, puncts and X (to catch hex and numbers lists).
I wished I could use a quick entropy computation, but these lines may have a high entropy too. But they have in some cases a fixed width format that is always the same and may be a clue to rely on: e.g. a formatted text where the format of each line itself is the same.
One example of data files (e.g. *.data) is in http://http.debian.net/debian/pool/main/a/ask/ask_1.0.1.orig.tar.gz
Other .hex files are found in the Linux kernel.
I have mixed feelings on data files: I want to scan them as they may contain licenses or clues AND I do not want to scan them if they are highly likely not to contain any clues.
For the copyright detection that seeks possible dates and date ranges (such as Copyright (c) 2000-2013 XXXX), long list of numbers are a hog for now.
There are other cases where scanning data files is not great especially when these are big files.
Or for instance the firmwares are mostly hex blobs in the kernel. There I still want to scan them even if these may be huge hex blob-like: they have the most byzantine licenses of all and are worth reporting.
So we need a better way to skip scanning certain files that are pure data.
And in these cases just issue a warning that they were not scanned fully and likely only scan the first and last hundred lines.
There are a couple patterns like binary-only data files which are easy to ignore.
Or text files with lines made of digits, puncts and X (to catch hex and numbers lists).
I wished I could use a quick entropy computation, but these lines may have a high entropy too. But they have in some cases a fixed width format that is always the same and may be a clue to rely on: e.g. a formatted text where the format of each line itself is the same.
One example of data files (e.g. *.data) is in http://http.debian.net/debian/pool/main/a/ask/ask_1.0.1.orig.tar.gz
Other .hex files are found in the Linux kernel.