scancode 21.8.4 starts with warnings

[tgodau]

Description

scancode 21.8.4 starts with warnings:

# scancode --version
/home/tgodau/venv3.8-scancode/lib/python3.8/site-packages/cluecode/copyrights.py:3355: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
/home/tgodau/venv3.8-scancode/lib/python3.8/site-packages/pygmars/lex.py:95: FutureWarning: Possible nested set at position 7
  re.compile(m, flags=re_flags).match
ScanCode version 21.8.4
#

How To Reproduce

I created a brand new pathon 3.8.10 virtual environment for the latest scancode version with

python3 -m venv ~/venv3.8-scancode

and switched to the new venv with

source ~/venv3.8-scancode/bin/activate

Because of some errors in a previous installation test, I installed the wheel-package:

pip install wheel

followed by the scancode installation:

pip install scancode-toolkit

The installation ran without problems (no warnings, no errors).

System configuration

OS: Ubuntu 20.04.3 LTS running in a Hyper-V VM
(until now I have scancode 21.6.7 running in a python 3.6.14 venv, which works very well)

[pombredanne]

Thank you for the report! That's annoying but this is harmless... it still needs to be fixed though.

[tgodau]

Morning Philippe,

the warning

/home/tgodau/venv3.8-scancode/lib/python3.8/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(

still exists in scancode-toolkit 30.1.0 for me (same environment as above)...

Regards, Thorsten

[sschuberth]

We're also seeing this in ScanCode 30.1.0:

/usr/local/scancode-toolkit-30.1.0/src/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 3 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
  File "/usr/local/scancode-toolkit-30.1.0/bin/scancode", line 33, in <module>
    sys.exit(load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')())
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/commoncode/cliutils.py", line 69, in main
    return click.Command.main(
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/scancode-toolkit-30.1.0/src/scancode/cli.py", line 451, in scancode
    success, _results = run_scan(
  File "/usr/local/scancode-toolkit-30.1.0/src/scancode/cli.py", line 887, in run_scan
    scan_success = run_scanners(
  File "/usr/local/scancode-toolkit-30.1.0/src/scancode/cli.py", line 1125, in run_scanners
    scan_success = scan_codebase(
  File "/usr/local/scancode-toolkit-30.1.0/src/scancode/cli.py", line 1236, in scan_codebase
    scan_timings) = next(scans)
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/click/_termui_impl.py", line 116, in __next__
    return next(iter(self))
  File "/usr/local/scancode-toolkit-30.1.0/lib/python3.8/site-packages/commoncode/cliutils.py", line 170, in generator
    yield from self.iter
  File "/usr/local/scancode-toolkit-30.1.0/src/scancode/pool.py", line 52, in wrap
    return func(self, timeout=timeout or 3600)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 863, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError

Might also be related to #2726.

[pombredanne]

Thanks. Warnings are innocuous but annoying alright!
But #2686 (comment) is another separate issue IMHO.

[sschuberth]

So, can you confirm that we're under no circumstances missing any licenses being detected because of this? I'm worried as I also see a multiprocessing.context.TimeoutError in the stack trace. Doesn't that mean that something eventually wasn't scanned?

[pombredanne]

@sschuberth re:

So, can you confirm that we're under no circumstances missing any licenses being detected because of this? I'm worried as I also see a multiprocessing.context.TimeoutError in the stack trace. Doesn't that mean that something eventually wasn't scanned?

Do you have a way to reproduce this? Which OS/arch do you run with this Python 3.8?
Is there a larger error trace? Does the JSON contain an error for a certain file?

[sschuberth]

Do you have a way to reproduce this?

Unfortunately not, as it happens with internal code.

Which OS/arch do you run with this Python 3.8?

It's the ORT Dockerfile, so based on Ubuntu Focal.

Is there a larger error trace? Does the JSON contain an error for a certain file?

Nope, that's all there is in the stack trace.

[DennisClark]

Out of date. Closing.