Can not extract archive file in pipline by extract_archives: worker-1 | INFO [error] normalize() argument 2 must be str, not bytes

[tpang-ameame]

Describe the bug
Scancode.io is deployed following documentation via docker compose.
Project can not extract archives from upload, cause empty result returned.
Error message indicates errors in extract_archives steps.
No error throw for whole pipline:

System configuration

• Which version of ScanCode.io are you running?
  v34.9.4
• Are you running the app using Docker?
  Yes
  Docker Compose version v2.24.5
  Docker version 24.0.2, build cb74dfc
• On which OS?
  Linux (centos 7.9.2009) 3.10.0-1160.92.1.el7.x86_64 x86_64 GNU/Linux
• What inputs are you using?
  .zip, .tar.gz
• Which pipeline are you running?
  scan_codebase, scan_single_package

To Reproduce
Steps to reproduce the behavior:

1. Create project and upload zip file
2. Execute pipline: scan_codebase
3. Empty result returned
   Error messages:
   [Severity] ERROR
   [Model]extract_archives
   [Description] normalize() argument 2 must be str, not bytes

Screenshots

[pombredanne]

Thanks for the detailed report 👍
Related #1578 (comment)

[pombredanne]

@tpang-ameame I have difficulties to reproduce. Do you have a link to a sample zip that has the issue?

[tpang-ameame]

Hi @pombredanne,
The issue can be always reproduced with different zip archives(small/large).
And after applying the version downgrade #1578 (comment), the issue is resolved. So I guess they share same root causes.

[tpang-ameame]

Meanwhile, I also hit following errors with some zip files:

2025-02-12 05:53:10.770 Pipeline [scan_codebase] starting
2025-02-12 05:53:10.774 Step [download_missing_inputs] starting
2025-02-12 05:53:10.778 Step [download_missing_inputs] completed in 0 seconds
2025-02-12 05:53:10.784 Step [copy_inputs_to_codebase_directory] starting
2025-02-12 05:53:10.819 Step [copy_inputs_to_codebase_directory] completed in 0 seconds
2025-02-12 05:53:10.824 Step [extract_archives] starting
2025-02-12 05:55:14.317 Step [extract_archives] completed in 123 seconds (2.1 minutes)
2025-02-12 05:55:14.326 Step [collect_and_create_codebase_resources] starting
2025-02-12 05:57:39.080 Step [collect_and_create_codebase_resources] completed in 145 seconds (2.4 minutes)
2025-02-12 05:57:39.087 Step [flag_empty_files] starting
2025-02-12 05:57:39.101 Step [flag_empty_files] completed in 0 seconds
2025-02-12 05:57:39.106 Step [flag_ignored_resources] starting
2025-02-12 05:57:43.548 Step [flag_ignored_resources] completed in 4 seconds
2025-02-12 05:57:43.554 Step [scan_for_application_packages] starting
2025-02-12 05:57:45.831 Progress: 10% (566/5651) ETA: 20 seconds
2025-02-12 05:57:46.888 Progress: 20% (1131/5651) ETA: 13 seconds
2025-02-12 05:57:47.908 Progress: 30% (1696/5651) ETA: 10 seconds
2025-02-12 05:57:48.721 Progress: 40% (2261/5651) ETA: 8 seconds
2025-02-12 05:57:49.602 Progress: 50% (2826/5651) ETA: 6 seconds
2025-02-12 05:57:50.507 Progress: 60% (3391/5651) ETA: 5 seconds
2025-02-12 05:57:51.411 Progress: 70% (3956/5651) ETA: 3 seconds
2025-02-12 05:57:52.340 Progress: 80% (4521/5651) ETA: 2 seconds
2025-02-12 05:57:53.301 Progress: 90% (5086/5651) ETA: 1 seconds
2025-02-12 05:57:54.159 Progress: 100% (5651/5651)
2025-02-12 05:57:54.418 Pipeline failed

'CodebaseResource' object has no attribute 'extracted_from'

Traceback:
  File "/opt/scancodeio/aboutcode/pipeline/__init__.py", line 199, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/scan_codebase.py", line 63, in scan_for_application_packages
    scancode.scan_for_application_packages(self.project, progress_logger=self.log)
  File "/opt/scancodeio/scanpipe/pipes/scancode.py", line 436, in scan_for_application_packages
    assemble_packages(project=project)
  File "/opt/scancodeio/scanpipe/pipes/scancode.py", line 493, in assemble_packages
    for item in items:
                ^^^^^
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/maven.py", line 123, in assemble
    yield from JavaJarManifestHandlerMixin.assemble(package_data, resource, codebase, package_adder)
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/models.py", line 1185, in assemble
    cls.assign_package_to_resources(
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/maven.py", line 158, in assign_package_to_resources
    models.DatafileHandler.assign_package_to_resources(
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/models.py", line 1225, in assign_package_to_resources
    archive_resource = resource.extracted_from(codebase)
                       ^^^^^^^^^^^^^^^^^^^^^^^

Does this indicate errors from extraction casue the scan failure? I can't share the zip for now cause it is for internal use.

[AyanSinhaMahapatra]

Thank you for reporting this @tpang-ameame, I was facing the same issue too from last week.
It seems the error was coming from the beautifulsoup4 library which we use for unicode text related functions in commoncode.text, and this is fixed by https://github.com/aboutcode-org/commoncode/releases/tag/v32.2.0 and #1583.

Could you try running scancode.io again from the latest main, and confirm this?

[AyanSinhaMahapatra]

Closing this as the fix is confirmed at #1578 (comment) and is also now released with https://github.com/aboutcode-org/scancode.io/releases/tag/v34.9.5.