scancode-toolkit-mini 30.1.0 - missing dependency

[eeisegn]

Description

When installing scancode-toolkit-mini in a clean system using pip it installs successfully, but produces this error when trying to execute it:

File "/Users/egans/PycharmProjects/scanoss.py/venv/lib/python3.8/site-packages/typecode/magic2.py", line 103, in load_lib_failover
ctypes.util.find_library('magic')
AttributeError: module 'ctypes' has no attribute 'util'

How To Reproduce

Install on a clean system where scancode has not been installed before.

pip install scancode-toolkit-mini scancode --help

If you install the full version (pip install scancode-toolkit), uninstall and then re-install mini (pip install scancode-toolkit-mini), it will work. When you do this, the following extra packages get installed:

extractcode-30.0.0 extractcode-7z-16.5.210531 extractcode-libarchive-3.5.1.210531 patch-1.16 scancode-toolkit-30.1.0 typecode-libmagic-5.39.210531

I believe the "mini" install is missing "typecode-libmagic" package dependency.

System configuration

• What OS are you running on? - MacOS & Linux
• What version of scancode-toolkit was used to generate the scan file? - 30.1.0, 21.8.4
• What installation method was used to install/run scancode? - pip

[pombredanne]

It is not missing anything, this is by design... the mini build is such that you are in control of dependencies using extras.
Can you elaborate what's is your use case?

[eeisegn]

If you take a clean machine with nothing from scancode previously installed, install the mini toolkit it will fail to run as it's missing the "typecode-libmagic" dependency.
So, I believe in order to make the scancode-toolkit-mini work, this dependency needs to be added?

[pombredanne]

So, I believe in order to make the scancode-toolkit-mini work, this dependency needs to be added?

Yes!

You need either: "typecode-libmagic" or another typecode plugin that "provides" libmagic... it could be actual system-provided libmagic such as:

• https://pypi.org/project/typecode-libmagic-from-sources/ to build it from sources
• https://pypi.org/project/typecode-libmagic-system-provided/ to get it from system packages

The question is why would you want the scancode-toolkit mini build? This is an advanced build where YOYO and meant only for special use where you can provide actual dependencies and have/want full control. I made these mostly to help support packaging in Linux distros starting with Debian

[VanL]

Hi @pombredanne, I just ran into this issue and it is not what you identified. Rather it is that the newest python does not automatically load ctypes.util when ctypes is imported. The fix is to modify typecode/magic2.py, where it says:

import ctypes
import glob

You change to:

import ctypes
import ctypes.util
import glob

and it works. You also may want to add an explanatory note to the error that no libmagic was found, suggesting that if they have installed libmagic in a nonstandard place, try setting TYPECODE_LIBMAGIC_PATH.