Skip to content

Commit a3dbeb1

Browse files
committed
Fix copyright detection for text-classified files and restore fixtures
This fixes a regression in numbered_text_lines() where valid text-bearing files were dropped when typecode reported contains_text == False, even though they were still text-classified and extractable. The early return prevented cluecode from detecting copyright notices in files misidentified by libmagic. We now allow text-classified files through the normal extraction path instead of rejecting them prematurely. This also restores corrupted ORT and junk-copyright fixture inputs that were causing false failures in the cluecode golden tests.
1 parent 6ba5908 commit a3dbeb1

4 files changed

Lines changed: 14 additions & 2 deletions

File tree

src/textcode/analysis.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,11 @@ def numbered_text_lines(
9393
logger_debug('numbered_text_lines: T.is_text_with_long_lines:', T.is_text_with_long_lines)
9494
logger_debug('numbered_text_lines: T.is_binary:', T.is_binary)
9595

96-
# TODO: we should have a command line to force digging inside binaries
97-
if not T.contains_text:
96+
# Some text-like files (notably source files misidentified by libmagic as
97+
# image/data with a "text" payload) still contain extractable content even
98+
# when `contains_text` is false. Allow text-classified files through to the
99+
# normal extraction path instead of dropping them prematurely.
100+
if not (T.contains_text or T.is_text):
98101
return iter([])
99102

100103
# Should we read this as some markup, pdf office doc, text or binary?
Binary file not shown.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
(c) 2007 James Turnbull <james@lovedthanlost.net>
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
what:
2+
- copyrights
3+
- holders
4+
copyrights:
5+
- "(c) 2007 James Turnbull <james@lovedthanlost.net>"
6+
holders:
7+
- "James Turnbull"
8+
notes: "ORT-derived fixture; see ORT test assets for provenance"

0 commit comments

Comments
 (0)