Pipeline failure

[pombredanne]

Using v31.0.0 and the scan_codebase pipeline with
https://download.qt.io/new_archive/qt/5.3/5.3.2/single/qt-everywhere-opensource-src-5.3.2.tar.xz
I got this log

2022-09-21 17:20:58.40 Pipeline [scan_codebase] starting
2022-09-21 17:20:58.40 Step [copy_inputs_to_codebase_directory] starting
2022-09-21 17:20:58.48 Step [copy_inputs_to_codebase_directory] completed in 0.07 seconds
2022-09-21 17:20:58.48 Step [extract_archives] starting
2022-09-21 17:21:37.20 Step [extract_archives] completed in 38.72 seconds
2022-09-21 17:21:37.21 Step [collect_and_create_codebase_resources] starting
2022-09-21 17:28:42.22 Step [collect_and_create_codebase_resources] completed in 425.01 seconds
2022-09-21 17:28:42.22 Step [tag_empty_files] starting
2022-09-21 17:28:42.27 Step [tag_empty_files] completed in 0.05 seconds
2022-09-21 17:28:42.27 Step [scan_for_application_packages] starting
2022-09-21 17:29:19.15 Pipeline failed

Task output

cursor "_django_curs_139684352849728_sync_9" does not exist


Traceback:
  File "/app/scanpipe/pipelines/__init__.py", line 115, in execute
    step(self)
  File "/app/scanpipe/pipelines/scan_codebase.py", line 93, in scan_for_application_packages
    scancode.scan_for_application_packages(self.project)
  File "/app/scanpipe/pipes/scancode.py", line 334, in scan_for_application_packages
    assemble_packages(project=project)
  File "/app/scanpipe/pipes/scancode.py", line 377, in assemble_packages
    for item in items:
  File "/usr/local/lib/python3.9/site-packages/packagedcode/models.py", line 952, in assemble
    cls.assign_package_to_resources(
  File "/usr/local/lib/python3.9/site-packages/packagedcode/build.py", line 54, in assign_package_to_resources
    models.DatafileHandler.assign_package_to_parent_tree(
  File "/usr/local/lib/python3.9/site-packages/packagedcode/models.py", line 1031, in assign_package_to_parent_tree
    cls.assign_package_to_resources(package, parent, codebase, package_adder)
  File "/usr/local/lib/python3.9/site-packages/packagedcode/models.py", line 1015, in assign_package_to_resources
    for res in resource.walk(codebase):
  File "/app/scanpipe/models.py", line 1762, in walk
    for subchild in child.walk(topdown=topdown):
  File "/app/scanpipe/models.py", line 1762, in walk
    for subchild in child.walk(topdown=topdown):
  File "/app/scanpipe/models.py", line 1762, in walk
    for subchild in child.walk(topdown=topdown):
  File "/app/scanpipe/models.py", line 1759, in walk
    for child in self.children().iterator():
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 512, in _iterator
    yield from iterable
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 87, in __iter__
    results = compiler.execute_sql(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
    cursor.close()

[tdruez]

@JonoYang This seems to be an issue within the assemble_packages step. Could you have a look at this one?

[JonoYang]

@pombredanne @tdruez

The issue is caused by the binary qt-everywhere-opensource-src-5.3.2\gnuwin32\bin\flex++.exe, where the ++ in the file name is causing issues with the regex matching when filtering for Resources in CodebaseResource.walk():

>>> r = p.codebaseresources.get(path="qt-everywhere-opensource-src-5.3.2.tar.xz-extract/qt-everywhere-opensource-src-5.3.2/gnuwin32/bin")
>>> for resource in r.walk():
...     print(resource.path)
qt-everywhere-opensource-src-5.3.2.tar.xz-extract/qt-everywhere-opensource-src-5.3.2/gnuwin32/bin/bison.exe
qt-everywhere-opensource-src-5.3.2.tar.xz-extract/qt-everywhere-opensource-src-5.3.2/gnuwin32/bin/bison.yacc
qt-everywhere-opensource-src-5.3.2.tar.xz-extract/qt-everywhere-opensource-src-5.3.2/gnuwin32/bin/flex++.exe
Traceback (most recent call last):
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.InvalidRegularExpression: invalid regular expression: quantifier operand invalid


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1395, in execute_sql
    cursor.execute(sql, params)
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: invalid regular expression: quantifier operand invalid


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/jono/nexb/src/scancode.io/scanpipe/models.py", line 1769, in walk
    for subchild in child.walk(topdown=topdown):
  File "/home/jono/nexb/src/scancode.io/scanpipe/models.py", line 1762, in walk
    children = list(self.children().iterator())
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/models/query.py", line 512, in _iterator
    yield from iterable
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/models/query.py", line 87, in __iter__
    results = compiler.execute_sql(
  File "/home/jono/nexb/src/scancode.io/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
    cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_139631293007680_sync_4" does not exist

[tdruez]

@JonoYang thanks for the fix!
I've added some exception handling to log those issues as ProjectErrors instead of failing the whole pipeline.
See #530 for a follow up issue.