Not usable with commoncode 21.1.14

[danielthieleog]

Description

When using new commoncode release 21.1.14 calling extractcode fails.
This was not the case with commoncode release 20.10.20.

Error message from extractcode

Traceback (most recent call last):
File "/app/venv-extractcode/bin/extractcode", line 5, in
from extractcode.cli import extractcode
File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/init.py", line 39, in
from commoncode.fileutils import fsencode
ImportError: cannot import name 'fsencode'

How To Reproduce

# Workdir is /app
pip install virtualenv \
    && virtualenv -p /usr/local/bin/python3.6 venv-extractcode \
    && . venv-extractcode/bin/activate \
    && pip install extractcode

virtualenv -p /usr/local/bin/python3.6 venv-extractcode
. venv-extractcode/bin/activate
    
/app/venv-extractcode/bin/extractcode

Error also occurs with python 3.9.1.

System configuration

Running on Linux in python:3.6 docker container.
Release 20.10 of extractcode.
Installed with pip.

[pombredanne]

@danielthieleog Thank you ++ for this. Let me publish a new version ASAP

[pombredanne]

and #3 should fix this alright

[danielthieleog]

With release 21.1.15 the error mentioned above is gone, but I get another error:

# /app/venv-extractcode/bin/extractcode /tmp/
Extracting archives...
Traceback (most recent call last):
  File "/app/venv-extractcode/bin/extractcode", line 8, in <module>
    sys.exit(extractcode())
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/commoncode/cliutils.py", line 87, in main
    **extra,
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/cli.py", line 184, in extractcode
    for xev in extraction_events:
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/_termui_impl.py", line 318, in generator
    for rv in self.iter:
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/api.py", line 42, in extract_archives
    ignore_pattern=ignore_pattern
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/extract.py", line 126, in extract
    for event in extract_files(location, kinds, recurse, ignore_pattern):
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/extract.py", line 184, in extract_files
    if not extractcode.archive.should_extract(loc, kinds, ignore_pattern):
AttributeError: module 'extractcode' has no attribute 'archive'

[JeroenKnoops]

Looks like something went wrong here: e872181#diff-e782914b32bc304259fc91067a094357eb7e617e1f5738f7ae7efca8d58f8581L192

[pombredanne]

@JeroenKnoops @danielthieleog Sorry for that mess! I wonder how this was not caught... a test is surely missing for the command line and top level functions (I think there used to be one in scancode-toolkit and should be brought back here).
Let me fix this.

[JeroenKnoops]

Now it's working with the new release: #4
Thanks very much!

[danielthieleog]

Does not work for me using the installation mentioned above under "How to reproduce":

# /app/venv-extractcode/bin/extractcode /tmp
Extracting archives...
Traceback (most recent call last):
  File "/app/venv-extractcode/bin/extractcode", line 8, in <module>
    sys.exit(extractcode())
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/commoncode/cliutils.py", line 87, in main
    **extra,
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/cli.py", line 184, in extractcode
    for xev in extraction_events:
  File "/app/venv-extractcode/lib/python3.6/site-packages/click/_termui_impl.py", line 318, in generator
    for rv in self.iter:
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/api.py", line 35, in extract_archives
    from extractcode.extract import extract
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/extract.py", line 34, in <module>
    import extractcode.archive
  File "/app/venv-extractcode/lib/python3.6/site-packages/extractcode/archive.py", line 30, in <module>
    from typecode import contenttype
  File "/app/venv-extractcode/lib/python3.6/site-packages/typecode/__init__.py", line 21, in <module>
    from typecode.contenttype import get_type
  File "/app/venv-extractcode/lib/python3.6/site-packages/typecode/contenttype.py", line 43, in <module>
    from typecode import magic2
  File "/app/venv-extractcode/lib/python3.6/site-packages/typecode/magic2.py", line 211, in <module>
    libmagic = load_lib()
  File "/app/venv-extractcode/lib/python3.6/site-packages/typecode/magic2.py", line 90, in load_lib
    'CRITICAL: libmagic DLL and is magic database are not installed. '
Exception: CRITICAL: libmagic DLL and is magic database are not installed. Unable to continue: you need to install a valid typecode-libmagic plugin with a valid and proper libmagic and magic DB available.

Additionally I installed:

pip install libmagic
pip install python-magic

but the error stays the same.

DLL means Windows? The installation image uses Linux.

[JeroenKnoops]

I've tested it on my mac and it works witn virtualenv.

What I found out is that I needed to add --force to the install command: pip install extractcode --force
Otherwise it would not install the latest version of the dependencies like commoncode.

[danielthieleog]

Same error with --force

# virtualenv -p /usr/local/bin/python3.6 venv-extractcode
created virtual environment CPython3.6.12.final.0-64 in 219ms
  creator CPython3Posix(dest=/app/venv-extractcode, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: PyYAML==5.4.1, attrs==20.3.0, beautifulsoup4==4.9.3, binaryornot==0.4.4, certifi==2020.12.5, cffi==1.14.4, chardet==4.0.0, click==7.1.2, commoncode==21.1.21, cryptography==3.3.1, extractcode==21.1.21, idna==2.10, importlib_metadata==3.4.0, intbitset==2.4.1, patch==1.16, pdfminer.six==20201018, pip==20.3.3, pluggy==0.13.1, plugincode==21.1.21, pycparser==2.20, requests==2.25.1, saneyaml==0.4, setuptools==51.3.3, six==1.15.0, sortedcontainers==2.3.0, soupsieve==2.1, text_unidecode==1.3, typecode==21.1.21, typing==3.6.6, typing_extensions==3.7.4.3, urllib3==1.26.2, wheel==0.36.2, zipp==3.4.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

[pombredanne]

@danielthieleog I have not yet added docs, but you would now need to install this way:

pip install extractcode[full]

Can you give it a try?

[pombredanne]

@danielthieleog re:

DLL means Windows? The installation image uses Linux.

That's used here as a generic term for dynamically linked library (e.g. AKA a "shared object")

[pombredanne]

@JeroenKnoops if you have some virtualenv that's already activated and has some installation, then the likely way is to use:
pip install --upgrade extractcode[full]

[danielthieleog]

@danielthieleog I have not yet added docs, but you would now need to install this way:

pip install extractcode[full]

Can you give it a try?

@pombredanne This works. Is it OK to keep this issue open until docs are adapted?

[JeroenKnoops]

@JeroenKnoops if you have some virtualenv that's already activated and has some installation, then the likely way is to use:
pip install --upgrade extractcode[full]

It works on my machine (and in a ci/cd pipeline on github action runners) so I'm happy! :)