Skip to content

Commit d5a26bb

Browse files
authored
Merge pull request #20 from nexB/use-env-var
Implement new library loading approach Contributed-by: Pierre Tardy <pierre.tardy@renault.com> Contributed-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
2 parents 45011e9 + 6395a4a commit d5a26bb

272 files changed

Lines changed: 13982 additions & 5561 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
# Ignore all Git auto CR/LF line endings conversions
2-
* binary
2+
* -text
3+
pyproject.toml export-subst

.travis.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ python:
1616
- "3.9"
1717

1818
# Scripts to run at install stage
19-
install: ./configure
19+
install: ./configure --dev
2020

2121
# Scripts to run at script stage
2222
script: tmp/bin/pytest -vvs -n 2

CHANGELOG.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,18 @@ Release notes
44
vNext
55
-----
66

7+
Version 21.5.30
8+
---------------
9+
10+
- Update vendored pygments to 2.9.0
11+
- Update commoncode to latest version
12+
- Use new libmagic configuration based on a plugin, and environment variable
13+
or the system path.
14+
- Only do content-based detection for programing language if there is no extension.
15+
- Remove Actionscript3 from programing language detection. This is not common
16+
enough and is the source of too many false positives.
17+
18+
719

820
Version 21.2.24
921
---------------

NOTICE

Lines changed: 5 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,8 @@
11
#
2-
# Copyright (c) nexB Inc. and others.
3-
# SPDX-License-Identifier: Apache-2.0
4-
#
5-
# Visit https://aboutcode.org and https://github.com/nexB/ for support and download.
2+
# Copyright (c) nexB Inc. and others. All rights reserved.
63
# ScanCode is a trademark of nexB Inc.
7-
#
8-
# Licensed under the Apache License, Version 2.0 (the "License");
9-
# you may not use this file except in compliance with the License.
10-
# You may obtain a copy of the License at
11-
#
12-
# http://www.apache.org/licenses/LICENSE-2.0
13-
#
14-
# Unless required by applicable law or agreed to in writing, software
15-
# distributed under the License is distributed on an "AS IS" BASIS,
16-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17-
# See the License for the specific language governing permissions and
18-
# limitations under the License.
4+
# SPDX-License-Identifier: Apache-2.0
5+
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
6+
# See https://github.com/nexB/typecode for support or download.
7+
# See https://aboutcode.org for more information about nexB OSS projects.
198
#

README.rst

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ macOS) and Pygments. It started as library in scancode-toolkit.
1212
Visit https://aboutcode.org and https://github.com/nexB/ for support and download.
1313

1414

15+
We run CI tests on:
16+
17+
- Travis https://travis-ci.org/github/nexB/typecode
18+
- Azure pipelines https://dev.azure.com/nexB/typecode/_build
19+
1520
To install this package with its full capability (where the binaries for
1621
libmagic are installed), use the `full` option::
1722

@@ -22,10 +27,39 @@ system, use the `minimal` option::
2227

2328
pip install typecode
2429

30+
In this case, you will need to provide a working libmagic and its database
31+
available in one of these ways:
32+
33+
- **a typecode-libmagic plugin**: See the standard ones at
34+
https://github.com/nexB/scancode-plugins/tree/main/builtins
35+
These can either bundle a libmagic library and its magic database or expose a
36+
system-installed libmagic.
37+
They do so by providing a plugin entry point as a ``scancode_location_provider``
38+
for ``typecode_libmagic`` which points to a callable that must return a mapping
39+
with these two keys:
40+
41+
- 'typecode.libmagic.dll': the absolute path to a libmagic DLL
42+
- 'typecode.libmagic.db': the absolute path to a libmagic 'magic.mgc' database
43+
44+
See for example:
45+
46+
- https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/typecode_libmagic-linux/setup.py#L42
47+
- https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/typecode_libmagic-linux/src/typecode_libmagic/__init__.py#L32
48+
49+
- **environment variables**:
50+
51+
- TYPECODE_LIBMAGIC_PATH: the absolute path to a libmagic DLL
52+
- TYPECODE_LIBMAGIC_DB_PATH: the absolute path to a libmagic 'magic.mgc' database
53+
54+
- **a system-installed libmagic and its database availale in the system PATH**:
55+
56+
57+
The supported libmagic version is 5.39.
58+
2559

2660
To set up the development environment::
2761

28-
source configure
62+
source configure --dev
2963

3064
To run unit tests::
3165

@@ -35,3 +69,13 @@ To clean up development environment::
3569

3670
./configure --clean
3771

72+
73+
To update Pygment to a newer vendored version use vendy:
74+
75+
- Update the version of pygments in ``pyproject.toml``
76+
- Run ``vendy``
77+
- Update the src/typecpde/pygments_lexers_mapping.py
78+
and src/typecode/pygments_lexers.py scripts accordingly, including their
79+
ABOUT files
80+
81+

azure-pipelines.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,39 +7,39 @@
77

88
jobs:
99

10-
- template: etc/ci/azure-linux.yml
10+
- template: etc/ci/azure-posix.yml
1111
parameters:
1212
job_name: ubuntu16_cpython
1313
image_name: ubuntu-16.04
1414
python_versions: ['3.6', '3.7', '3.8', '3.9']
1515
test_suites:
1616
all: tmp/bin/pytest -vvs
1717

18-
- template: etc/ci/azure-linux.yml
18+
- template: etc/ci/azure-posix.yml
1919
parameters:
2020
job_name: ubuntu18_cpython
2121
image_name: ubuntu-18.04
2222
python_versions: ['3.6', '3.7', '3.8', '3.9']
2323
test_suites:
2424
all: tmp/bin/pytest -n 2 -vvs
2525

26-
- template: etc/ci/azure-linux.yml
26+
- template: etc/ci/azure-posix.yml
2727
parameters:
2828
job_name: ubuntu20_cpython
2929
image_name: ubuntu-20.04
3030
python_versions: ['3.6', '3.7', '3.8', '3.9']
3131
test_suites:
3232
all: tmp/bin/pytest -n 2 -vvs
3333

34-
- template: etc/ci/azure-mac.yml
34+
- template: etc/ci/azure-posix.yml
3535
parameters:
3636
job_name: macos1014_cpython
3737
image_name: macos-10.14
3838
python_versions: ['3.6', '3.7', '3.8', '3.9']
3939
test_suites:
4040
all: tmp/bin/pytest -n 2 -vvs
4141

42-
- template: etc/ci/azure-mac.yml
42+
- template: etc/ci/azure-posix.yml
4343
parameters:
4444
job_name: macos1015_cpython
4545
image_name: macos-10.15

configure

Lines changed: 145 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,169 @@
11
#!/usr/bin/env bash
22
#
3-
# Copyright (c) nexB Inc. and others.
3+
# Copyright (c) nexB Inc. and others. All rights reserved.
44
# SPDX-License-Identifier: Apache-2.0
5+
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
6+
# See https://github.com/nexB/ for support or download.
7+
# See https://aboutcode.org for more information about nexB OSS projects.
58
#
69

710
set -e
811
#set -x
912

10-
# source this script for a basic setup and configuration for local development
13+
################################
14+
# A configuration script to set things up:
15+
# create a virtualenv and install or update thirdparty packages.
16+
# Source this script for initial configuration
17+
# Use configure --help for details
18+
#
19+
# This script will search for a virtualenv.pyz app in etc/thirdparty/virtualenv.pyz
20+
# Otherwise it will download the latest from the VIRTUALENV_PYZ_URL default
21+
################################
22+
CLI_ARGS=$1
1123

12-
CONFIGURE_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
24+
################################
25+
# Defaults. Change these variables to customize this script
26+
################################
1327

28+
# Requirement arguments passed to pip and used by default or with --dev.
29+
REQUIREMENTS="--editable .[full]"
30+
DEV_REQUIREMENTS="--editable .[full,testing]"
1431

15-
if [[ "$1" == "--clean" ]]; then
16-
rm -rf "$CONFIGURE_ROOT_DIR/tmp"
17-
exit
32+
# where we create a virtualenv
33+
VIRTUALENV_DIR=tmp
34+
35+
# Cleanable files and directories with the --clean option
36+
CLEANABLE="
37+
build
38+
tmp"
39+
40+
# extra arguments passed to pip
41+
PIP_EXTRA_ARGS=" "
42+
43+
# the URL to download virtualenv.pyz if needed
44+
VIRTUALENV_PYZ_URL=https://bootstrap.pypa.io/virtualenv.pyz
45+
################################
46+
47+
48+
################################
49+
# Current directory where this script lives
50+
CFG_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
51+
CFG_BIN_DIR=$CFG_ROOT_DIR/$VIRTUALENV_DIR/bin
52+
53+
54+
################################
55+
# Set the quiet flag to empty if not defined
56+
if [[ "$CFG_QUIET" == "" ]]; then
57+
CFG_QUIET=" "
1858
fi
1959

2060

21-
if [[ "$PYTHON_EXE" == "" ]]; then
22-
PYTHON_EXE=python3
61+
################################
62+
# find a proper Python to run
63+
# Use environment variables or a file if available.
64+
# Otherwise the latest Python by default.
65+
if [[ "$PYTHON_EXECUTABLE" == "" ]]; then
66+
# check for a file named PYTHON_EXECUTABLE
67+
if [ -f "$CFG_ROOT_DIR/PYTHON_EXECUTABLE" ]; then
68+
PYTHON_EXECUTABLE=$(cat "$CFG_ROOT_DIR/PYTHON_EXECUTABLE")
69+
else
70+
PYTHON_EXECUTABLE=python3
71+
fi
2372
fi
2473

2574

26-
function setup {
27-
# create a virtualenv on Python
28-
mkdir -p $CONFIGURE_ROOT_DIR/tmp
29-
wget -O $CONFIGURE_ROOT_DIR/tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz
30-
$PYTHON_EXE $CONFIGURE_ROOT_DIR/tmp/virtualenv.pyz $CONFIGURE_ROOT_DIR/tmp
31-
source $CONFIGURE_ROOT_DIR/tmp/bin/activate
32-
$CONFIGURE_ROOT_DIR/tmp/bin/pip install --upgrade pip virtualenv setuptools wheel
75+
################################
76+
cli_help() {
77+
echo An initial configuration script
78+
echo " usage: ./configure [options]"
79+
echo
80+
echo The default is to configure for regular use. Use --dev for development.
81+
echo
82+
echo The options are:
83+
echo " --clean: clean built and installed files and exit."
84+
echo " --dev: configure the environment for development."
85+
echo " --help: display this help message and exit."
86+
echo
87+
echo By default, the python interpreter version found in the path is used.
88+
echo Alternatively, the PYTHON_EXECUTABLE environment variable can be set to
89+
echo configure another Python executable interpreter to use. If this is not
90+
echo set, a file named PYTHON_EXECUTABLE containing a single line with the
91+
echo path of the Python executable to use will be checked last.
92+
set +e
93+
exit
3394
}
3495

3596

36-
setup
97+
clean() {
98+
# Remove cleanable file and directories and files from the root dir.
99+
echo "* Cleaning ..."
100+
for cln in $CLEANABLE;
101+
do rm -rf "${CFG_ROOT_DIR:?}/${cln:?}";
102+
done
103+
set +e
104+
exit
105+
}
37106

38-
$CONFIGURE_ROOT_DIR/tmp/bin/pip install -e .[testing] -e .[full]
39107

40-
if [ -f "$CONFIGURE_ROOT_DIR/tmp/bin/activate" ]; then
41-
source "$CONFIGURE_ROOT_DIR/tmp/bin/activate"
42-
fi
108+
create_virtualenv() {
109+
# create a virtualenv for Python
110+
# Note: we do not use the bundled Python 3 "venv" because its behavior and
111+
# presence is not consistent across Linux distro and sometimes pip is not
112+
# included either by default. The virtualenv.pyz app cures all these issues.
113+
114+
VENV_DIR="$1"
115+
if [ ! -f "$CFG_BIN_DIR/python" ]; then
116+
117+
mkdir -p "$CFG_ROOT_DIR/$VENV_DIR"
118+
119+
if [ -f "$CFG_ROOT_DIR/etc/thirdparty/virtualenv.pyz" ]; then
120+
VIRTUALENV_PYZ="$CFG_ROOT_DIR/etc/thirdparty/virtualenv.pyz"
121+
else
122+
VIRTUALENV_PYZ="$CFG_ROOT_DIR/$VENV_DIR/virtualenv.pyz"
123+
wget -O "$VIRTUALENV_PYZ" "$VIRTUALENV_PYZ_URL"
124+
fi
125+
126+
$PYTHON_EXECUTABLE "$VIRTUALENV_PYZ" \
127+
--wheel embed --pip embed --setuptools embed \
128+
--seeder pip \
129+
--never-download \
130+
--no-periodic-update \
131+
--no-vcs-ignore \
132+
$CFG_QUIET \
133+
"$CFG_ROOT_DIR/$VENV_DIR"
134+
fi
135+
}
136+
137+
138+
install_packages() {
139+
# install requirements in virtualenv
140+
# note: --no-build-isolation means that pip/wheel/setuptools will not
141+
# be reinstalled a second time and reused from the virtualenv and this
142+
# speeds up the installation.
143+
# We always have the PEP517 build dependencies installed already.
144+
145+
"$CFG_BIN_DIR/pip" install \
146+
--upgrade \
147+
--no-build-isolation \
148+
$CFG_QUIET \
149+
$PIP_EXTRA_ARGS \
150+
$1
151+
}
152+
153+
154+
################################
155+
# Main command line entry point
156+
CFG_DEV_MODE=0
157+
CFG_REQUIREMENTS=$REQUIREMENTS
158+
159+
case "$CLI_ARGS" in
160+
--help) cli_help;;
161+
--clean) clean;;
162+
--dev) CFG_REQUIREMENTS="$DEV_REQUIREMENTS" && CFG_DEV_MODE=1;;
163+
esac
164+
165+
create_virtualenv "$VIRTUALENV_DIR"
166+
install_packages "$CFG_REQUIREMENTS"
167+
. "$CFG_BIN_DIR/activate"
43168

44169
set +e

0 commit comments

Comments
 (0)