copyright scan error

[dd-jy]

Description

Please leave a brief description of the bug or feature request:

Copyright scan error occured.

{
      "path": "source/.github/workflows/test.yaml",
      "type": "file",
      "detected_license_expression": null,
      "detected_license_expression_spdx": null,
      "license_detections": [],
      "license_clues": [],
      "percentage_of_license_text": 0,
      "copyrights": [],
      "holders": [],
      "authors": [],
      "scan_errors": [
        "ERROR: for scanner: copyrights:\nERROR: Unknown error:\nTraceback (most recent call last):\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/scancode/interrupt.py\", line 91, in interruptible\n    return NO_ERROR, func(*(args or ()), **(kwargs or {}))\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/scancode/api.py\", line 71, in get_copyrights\n    copyrights, holders, authors = Detection.split(detections, to_dict=True)\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/cluecode/copyrights.py\", line 434, in split\n    for detection in detections:\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/cluecode/copyrights.py\", line 110, in detect_copyrights\n    numbered_lines = numbered_text_lines(location, demarkup=demarkup)\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/textcode/analysis.py\", line 116, in numbered_text_lines\n    if demarkup and markup.is_markup(location):\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/textcode/markup.py\", line 67, in is_markup\n    return is_markup_text(start)\n  File \"/home/oscsystem/venv3/lib/python3.10/site-packages/textcode/markup.py\", line 72, in is_markup_text\n    if text.startswith('<'):\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\n"
      ]

How To Reproduce

Tell us how to reproduce the issue.

1. Download the check source code : git clone https://github.com/go-check/check
2. Command : scancode -l -c --json-pp out.json check

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? (Windows/MacOS/Linux) : Linux 22.04 / python3.10.12
• What version of scancode-toolkit was used to generate the scan file? : 32.2.1
• What installation method was used to install/run scancode? (pip/source download/other) : pip

[watschi]

Can confirm this issue for Java source files with license comments at the start of the file, e.g. https://github.com/apache/guacamole-client/blob/main/guacamole/src/main/java/org/apache/guacamole/GuacamoleSession.java
This produces 400k+ errors for a scan of the guacamole-client repository incl. dependencies.

How To Reproduce

1. Download the above mentioned file
2. Command: scancode -cli --json out.json ./GuacamoleSession.java

ERROR: for scanner: copyrights:
ERROR: Unknown error:
Traceback (most recent call last):
  File \"/usr/local/lib/python3.12/site-packages/scancode/interrupt.py\", line 91, in interruptible
    return NO_ERROR, func(*(args or ()), **(kwargs or {}))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/scancode/api.py\", line 71, in get_copyrights
    copyrights, holders, authors = Detection.split(detections, to_dict=True)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/cluecode/copyrights.py\", line 465, in split
    for detection in detections:
                     ^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/cluecode/copyrights.py\", line 127, in detect_copyrights
    numbered_lines = list(numbered_text_lines(location, demarkup=True))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/textcode/analysis.py\", line 115, in numbered_text_lines
    if demarkup and markup.is_markup(location):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/textcode/markup.py\", line 76, in is_markup
    return is_markup_text(start)
           ^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.12/site-packages/textcode/markup.py\", line 83, in is_markup_text
    if text.startswith(\"<\"):
       ^^^^^^^^^^^^^^^^^^^^
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

System configuration

What OS are you running on? : Rocky Linux release 8.9 (Green Obsidian) / Python 3.12.8
What version of scancode-toolkit was used to generate the scan file? : 32.3.2
What installation method was used to install/run scancode? (pip/source download/other) : pip

[watschi]

Seems like this was introduced with beautifulsoup 4.13, from the Changelog:

• UnicodeDammit.markup is now always a bytestring representing the
  original markup (sans BOM), and UnicodeDammit.unicode_markup is
  always the converted Unicode equivalent of the original
  markup. Previously, UnicodeDammit.markup was treated inconsistently
  and would often end up containing Unicode. UnicodeDammit.markup was
  not a documented attribute, but if you were using it, you probably
  want to switch to using .unicode_markup instead.

If UnicodeDammit(s).unicode_markup is used here instead of UnicodeDammit(s).markup, a unicode string is returned:

[AyanSinhaMahapatra]

@dd-jy thanks for reporting this and @watschi thanks for looking into the cause, this is fixed at commoncode and should be added here with #4149