Dockerfile (alpine based) + some quick ideas/suggestions

[roscopecoltran]

Hi guys,

Hope you are all well !

I found your great repo while I was working on a personal project aiming to do, also, some code analysis with a distributed bot, working as a virtual agent to build a semi-structured database of meta informations about some open source projects (mainly SQL and graph based).

Context:
The goal is to enrich automatically, with a virtual assistant/bot, some of my starred repos by 'left joining', or apply some graphql queries, on some data like, known frameworks detection, github stats or github trends, in order to extract more meta data from some of my starred repositories.

In a nutshell, it is all about to have a more convenient local search engines on my starred repos, create a domain specific topic modeling api. By detecting some patterns, I want to manage a dynamic tree of events that could be dedicated to some sub-tasks like to generate Docker/Docker-Compose files from a database a snippets if some dependencies are detected or matched.

examples of pattern detection to build dynamic dockerfiles:

• requirements.txt -> append pyp-pip to apk packages to install in a Alpine based dockerfile, but could do the mapping for an Ubuntu based dockerfile for apt package label, python-pip.
  • Docker alpine: apk --no-cache --no-progress --update pyp-pip
  • Docker ubuntu: apt-get install python-pip
• CMakeLists.txt -> add cmake, add some additional helpers for building C/C++ based projects.

1. Deploy local instances with Docker/Docker Compose

For some tasks, it sounds clear that some content lexer, grep, parser would provide faster responses and results; dependencies scanning, licences; so that s how I found your projects, and that really cool :-)

So first of all, as a dev ops ^^, I was wondering if it would not be easier to bundle scancode into some docker/docker-compose files, of course alpine based in order to keep the size of containers reasonable.

2. Some suggestions/features

So far, I started to give a try to your projects, and it sounds that I will create a fork with some personal ideas/features, close to want the context mentioned above, and wanted to share it with you, so any feedback or experience sharing is tremendously welcomed :-)

- Administration panel

• in order to refresh, modernize the server side of scancode with a modern admin ui (reactjs or angularjs)
• https://github.com/aio-libs/aiohttp_admin (Aiohttp + NGAdmin), but you can use some other admin boilerplates from the same authors
• https://github.com/marmelab/admin-on-rest
• https://github.com/marmelab/admin-on-rest-graphql-demo

- Topic modelling

• https://github.com/wowdd1/xlinkBook
• https://github.com/leonbrain/FYP_knowledgeGraph
• https://github.com/InformatiCup/InformatiCup2017/blob/master/results/README.md (3 winners projects for github classifiers)
• https://github.com/src-d/blog/blob/master/content/post/github_topic_modeling.md
  • docker run srcd/github_topics apache/spark (replace apache/spark with any GitHub repository you wish to analyze).

- Code parsing/search

• https://github.com/src-d/enry (aim to be faster than the linguist)
• https://github.com/porter-io/tag-github (for custom mappings and regex rules)
• example of mapping brands, keywords: https://github.com/porter-io/tagg-python/blob/master/tagg/default_defs.json
• https://github.com/google/zoekt (Sounds promising)
• https://github.com/junkblocker/codesearch

-Starred Github managers:

• https://github.com/hoop33/limo
• https://voblet.com/github

3. Questions related to your roadmap:

Last question, do you plan to migrate to Python3 your stack of scripts ? Is it something that you have on the roadmap and would like to have a community efforts ?

Ps. sorry for the long post, but I was really inspired by your work ^^, so thanks for reading it all :-)

Please have a great day !

Cheers,
Richard

[pombredanne]

@roscopecoltran this sounds like an awesome plan!

Having a Dockerfile would be great indeed.

Some other notes OTH:

• about Python3: yes the plan is to port to Python... and this should happen pretty soon after the release of v2.0.0. Help is definitely welcomed. The plan was to eventually support only Python 3.6 and up and the question is whether or not to continue support Python 2.

• something that is likely to help you a lot is the upcoming plugin architecture by @yashdsaraf

[roscopecoltran]

Hey,

Thanks for the reply !

It would be awesome to have a flow based processing for starting concurrent tasks on the code. defined in some human readable yaml files ^^.

Also, for any meta based additional info, about a project, I would recommend to use searx as it can flexible on the scope of metasearch to gather for public/web related info, if u want to build a context around the code audit.

Have a good week-end :-)

Cheers,
Richard

[pombredanne]

btw the ticket for Python 3 is #295 and also #442

[roscopecoltran]

Bonjour Philippe, :-)

Hope you are all well !

I am bundling scancode into a docker alpine container but some errors occured.

I think that some tweaks are required while running the configure script. Mainly, it would be great to use fallbacks to find the library outside the scancode project, like operating a search in standard locations, and keeping the musl order, eg. paths = ['/lib', '/usr/local/lib', '/usr/lib'], or maybe to trigger a build event if the pre-compiled lib failed to be loaded.

In our case, I add such error message while trying to link the libmagic2.so shared library, the configure scripts fails with "__snprintf_chk: symbol not found"; probably due to musl-dev or glibc).

From my understanding, these functions are used to find and map the pre-compiled libmagic shared lib.

• https://github.com/nexB/scancode-toolkit/blob/develop/src/commoncode/command.py#L321-L340
• https://github.com/nexB/scancode-toolkit/blob/develop/src/typecode/magic2.py#L192-L211

The easier solution would be to use an ubuntu based container, but the size of Ubuntu container is such a none-sense for me:

❯ docker images | awk '{print $1"\t"$2"\t"$7" "$8}'
REPOSITORY  TAG SIZE
sample_alpine   latest  167.6 MB
sample_ubuntu   latest  447.8 MB
<none>  <none>  187.9 MB
ubuntu  latest  187.9 MB
redis   latest  109.3 MB
mongo   latest  261.6 MB
golang  latest  709.5 MB
alpine  latest  5.249 MB

Refs used:

• Alpine + GLIBC: https://github.com/frol/docker-alpine-glibc/blob/master/Dockerfile
• Alpine + MiniConda3: https://github.com/frol/docker-alpine-miniconda3
• Alpine + OracleJDK8: https://github.com/frol/docker-alpine-oraclejdk8
• Alpine + Machine Learning: https://github.com/frol/docker-alpine-python-machinelearning
• Alpine + Musl find patch: https://github.com/JJTC-Docker/python/blob/master/musl-find_library.patch
• Find package name to add through apk manager based on file names : https://github.com/jessfraz/apk-file

Waiting for your input/point of view on that question. :-)

Cheers,
Richard

[pombredanne]

@roscopecoltran this is effectively a todo: the prebuilt binaries need to be unbundled ... this is tracked in #469

[znerd]

Here's a simple Dockerfile based on Alpine Linux 3.8 and Python 2.7.15:

FROM python:2.7.15-alpine
MAINTAINER Ernst de Haan "ernst.dehaan@mindcurv.com"

ARG SCANCODE_VERSION

RUN apk add build-base libxml2-dev libxslt-dev linux-headers
RUN pip install scancode-toolkit==${SCANCODE_VERSION}

CMD [ "/usr/local/bin/scancode" ]

Source can also be found here:

• https://bitbucket.org/mindcurv/scancode-toolkit-docker/src/master/Dockerfile

And the container here:

• https://hub.docker.com/r/mindcurv/scancode-toolkit/

Here's how to build it with a version tag:

docker build -t scancode-toolkit:latest -t scancode-toolkit:2.2.1 --build-arg SCANCODE_VERSION=2.2.1 .

[znerd]

Hmm, while a simple scancode --help works OK:

docker run -it mindcurv/scancode-toolkit:2.2.1 scancode --help

…a real scan doesn't work yet. I'm running into (apparently) the same issue as @roscopecoltran :

$ docker run -it mindcurv/scancode-toolkit:2.2.1 scancode --format json . scancode_result.json
Scanning files for: licenses, copyrights, packages with 1 process(es)...
Building license detection index...Traceback (most recent call last):
  File "/usr/local/bin/scancode", line 11, in <module>
    sys.exit(scancode())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scancode/utils.py", line 74, in main
    standalone_mode=standalone_mode, **extra)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scancode/cli.py", line 490, in scancode
    pre_scan_plugins=pre_scan_plugins)
  File "/usr/local/lib/python2.7/site-packages/scancode/cli.py", line 572, in scan
    get_index(False)
  File "/usr/local/lib/python2.7/site-packages/licensedcode/cache.py", line 188, in get_index
    _LICENSES_INDEX = get_or_build_index_through_cache()
  File "/usr/local/lib/python2.7/site-packages/licensedcode/cache.py", line 108, in get_or_build_index_through_cache
    from licensedcode.index import LicenseIndex
  File "/usr/local/lib/python2.7/site-packages/licensedcode/index.py", line 47, in <module>
    from licensedcode import match
  File "/usr/local/lib/python2.7/site-packages/licensedcode/match.py", line 36, in <module>
    from licensedcode import query
  File "/usr/local/lib/python2.7/site-packages/licensedcode/query.py", line 32, in <module>
    import typecode
  File "/usr/local/lib/python2.7/site-packages/typecode/__init__.py", line 27, in <module>
    from typecode.contenttype import get_type
  File "/usr/local/lib/python2.7/site-packages/typecode/contenttype.py", line 47, in <module>
    from typecode import magic2
  File "/usr/local/lib/python2.7/site-packages/typecode/magic2.py", line 221, in <module>
    libmagic = load_lib()
  File "/usr/local/lib/python2.7/site-packages/typecode/magic2.py", line 214, in load_lib
    lib = ctypes.CDLL(magic_so)
  File "/usr/local/lib/python2.7/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error relocating /usr/local/lib/python2.7/site-packages/typecode/bin/linux-64/lib/libmagic.so: __snprintf_chk: symbol not found

I will do some more research.

[znerd]

I've created a 2nd Dockerfile, this time based on Debian Stretch. That seems to work.

Stuff is over here:

• https://bitbucket.org/mindcurv/scancode-toolkit-docker/
• https://hub.docker.com/r/mindcurv/scancode-toolkit/

[pombredanne]

@znerd Thanks you for figuring all this out and sorry for some of the troubles: keep me posted as you progress!

At the moment, an Alpine container would require a bit of manual work as we there are some bundled, pre-built binaries that would need to be rebuilt by hand for this to work with an Alpine-style static build context.

Things should work fine with Debian/Ubuntu/CentOS/Fedora/Suse and similar

Here are two other examples: https://github.com/clearlydefined/tool-images/blob/0392771a408dbfb2ab5fcd88f702c43f207aa4ce/scancode/Dockerfile

https://github.com/clearlydefined/crawler/blob/1567d78abb7c1ee00c4ef89129ae9f1c56c92df4/Dockerfile

I am not sure which version of Scancode you used but I would suggest using the latest 2.9.x pre v3 releases.

[sschuberth]

I'm currently experimenting with a minimal Dockerfile that already has glibc installed:

FROM frolvlad/alpine-python2

RUN pip install scancode-toolkit

But building the image gives

Collecting extractcode-libarchive (from scancode-toolkit)
ERROR: Could not find a version that satisfies the requirement extractcode-libarchive (from scancode-toolkit) (from versions: none)
ERROR: No matching distribution found for extractcode-libarchive (from scancode-toolkit)

@pombredanne, any idea what the issue is?

[pombredanne]

@sschuberth sorry for the late reply...
ScanCode has a dep on extractcode-libarchive and this contains a prebuilt native that may not be happy with an Alpine static setup.
What is likely is that the glibc provided there may not match?

Could you try to use a release archive with this Alpine setup instead to eliminate some moving parts?

Just our if curiosity is using Alpine really proving such big benefits for the pain it brings?

[sschuberth]

What is likely is that the glibc provided there may not match?

What are the criteria by which glibc is matched?

Could you try to use a release archive with this Alpine setup instead to eliminate some moving parts?

Will try.

Just our if curiosity is using Alpine really proving such big benefits for the pain it brings?

Well, actually ScanCode is the first project I run into which resists to easily run on Alpine. But yes, image-size wise I believe it's worth the effort.

[pombredanne]

@sschuberth

What are the criteria by which glibc is matched?

I have no idea :D

that said, it (making things work on Alpine) could be something that could may be dealt with in the GSoC project of @aj4ayushjain ?