TypeError while scanning file (PDF)

[vw-anton]

Description

When scanning a PDF from https://github.com/hopding/pdf-lib (rev 4beebbef7a261ed0be089bb1d63a5b6e28cc488a) we receive:

TypeError while scanning file 'assets/pdfs/with_invalid_objects.pdf' (int() argument must be a string, a bytes-like object or a real number, not 'PSKeyword'

How To Reproduce

ScanCode 32.1.0 via ORT (22.5.0)

 config:
      ScanCode:
        options:
          commandLine: "--copyright --license --license-text --info --strip-root --timeout\
            \ 600"
          parseLicenseExpressions: "true"

System configuration

• What OS are you running on? (Windows/MacOS/Linux)
  • Linux
• What version of scancode-toolkit was used to generate the scan file?
  • 32.1.0
• What installation method was used to install/run scancode? (pip/source download/other)
  • pip in ORT

[pombredanne]

@vw-anton Thanks for the report! This is likely a bug in the underlying PDF library. Do you have a full stack trace?

[vw-anton]

From ORT we only received

  issues:
      - timestamp: "2024-06-06T08:26:01.748905945Z"
        source: "ScanCode"
        message: "ERROR: TypeError while scanning file 'assets/pdfs/with_invalid_objects.pdf'\
          \ (int() argument must be a string, a bytes-like object or a real number,\
          \ not 'PSKeyword')."
        severity: "ERROR"

I ran this via scancode.io but in the exported JSON I cannot see any indicator, the the console logs too.

[stefan6419846]

Running https://github.com/Hopding/pdf-lib/blob/93dd36e85aa659a3bca09867d2d8fac172501fbe/assets/pdfs/with_invalid_objects.pdf through scancode -l --license-text --license-text-diagnostics --yaml error.yml with_invalid_objects.pdf yields:

headers:
    -   tool_name: scancode-toolkit
        tool_version: 32.1.0
        options:
            input:
                - with_invalid_objects.pdf
            --license: yes
            --license-text: yes
            --license-text-diagnostics: yes
            --yaml: error.yml
        notice: |
            Generated with ScanCode and provided on an "AS IS" BASIS, WITHOUT WARRANTIES
            OR CONDITIONS OF ANY KIND, either express or implied. No content created from
            ScanCode should be considered or used as legal advice. Consult an Attorney
            for any legal advice.
            ScanCode is a free software code scanning tool from nexB Inc. and others.
            Visit https://github.com/nexB/scancode-toolkit/ for support and download.
        start_timestamp: '2024-06-06T192652.204458'
        end_timestamp: '2024-06-06T192654.816752'
        output_format_version: 3.1.0
        duration: '2.6123127937316895'
        message:
        errors:
            - 'Path: with_invalid_objects.pdf'
        warnings: []
        extra_data:
            system_environment:
                operating_system: linux
                cpu_architecture: 64
                platform: Linux-6.5.0-10040-tuxedo-x86_64-with-glibc2.35
                platform_version: '#44 SMP PREEMPT_DYNAMIC Wed May  8 17:36:39 UTC 2024'
                python_version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
            spdx_license_list_version: '3.23'
            files_count: 1
license_detections: []
files:
    -   path: with_invalid_objects.pdf
        type: file
        detected_license_expression:
        detected_license_expression_spdx:
        license_detections: []
        license_clues: []
        percentage_of_license_text: '0'
        scan_errors:
            - |
                ERROR: for scanner: licenses:
                ERROR: Unknown error:
                Traceback (most recent call last):
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 991, in read_xref_from
                    (pos, token) = parser.nexttoken()
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/psparser.py", line 526, in nexttoken
                    self.fillbuf()
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/psparser.py", line 239, in fillbuf
                    raise PSEOF("Unexpected EOF")
                pdfminer.psparser.PSEOF: Unexpected EOF

                During handling of the above exception, another exception occurred:

                Traceback (most recent call last):
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 722, in __init__
                    self.read_xref_from(parser, pos, self.xrefs)
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 993, in read_xref_from
                    raise PDFNoValidXRef("Unexpected EOF")
                pdfminer.pdfdocument.PDFNoValidXRef: Unexpected EOF

                During handling of the above exception, another exception occurred:

                Traceback (most recent call last):
                  File "/tmp/venv/lib/python3.10/site-packages/scancode/interrupt.py", line 91, in interruptible
                    return NO_ERROR, func(*(args or ()), **(kwargs or {}))
                  File "/tmp/venv/lib/python3.10/site-packages/scancode/api.py", line 200, in get_licenses
                    for detection in detections:
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/detection.py", line 1926, in detect_licenses
                    license_matches = index.match(
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/index.py", line 928, in match
                    qry = query.build_query(
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/query.py", line 138, in build_query
                    qry = Query(
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/query.py", line 293, in __init__
                    self.tokenize_and_build_runs(
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/query.py", line 542, in tokenize_and_build_runs
                    self._tokenize_and_build_runs(tokens_by_line, line_threshold)
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/query.py", line 598, in _tokenize_and_build_runs
                    for tokens in tokens_by_line:
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/query.py", line 416, in tokens_by_line
                    for line_num, line in qlines:
                  File "/tmp/venv/lib/python3.10/site-packages/licensedcode/tokenize.py", line 45, in query_lines
                    numbered_lines = numbered_text_lines(
                  File "/tmp/venv/lib/python3.10/site-packages/textcode/analysis.py", line 102, in numbered_text_lines
                    if T.is_pdf and T.is_pdf_with_text:
                  File "/tmp/venv/lib/python3.10/site-packages/typecode/contenttype.py", line 594, in is_pdf_with_text
                    doc = PDFDocument(parser)
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 727, in __init__
                    newxref.load(parser)
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 241, in load
                    (_, obj) = parser.nextobject()
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/psparser.py", line 656, in nextobject
                    self.do_keyword(pos, token)
                  File "/tmp/venv/lib/python3.10/site-packages/pdfminer/pdfparser.py", line 79, in do_keyword
                    (objid, genno) = (int(objid), int(genno))  # type: ignore[arg-type]
                TypeError: int() argument must be a string, a bytes-like object or a real number, not 'PSKeyword'

This file is (intentionally) broken. In this case, all tools I have tried complain about issues. (pypdf does not fail on this file in non-strict mode, but such edge cases are generally handled differently by each library anyway.)

[stefan6419846]

Possible duplicate of #2579.