Make PyPI installation work on Python 2 #1833 by pombredanne · Pull Request #1835 · aboutcode-org/scancode-toolkit

pombredanne · 2019-11-19T09:17:54Z

This PR is a fix for #1833 to properly install plugins on Python 2.
There is in fact some updates as plugins are moved to their own repos, we add new native wheels for windows64 and update the deps. And #335 to support Windows64.
Along the way there are a few other things fixed such as #1971 and extractcode support for 7z on case insensitive macOS filesystems

It also tries to speed up the CI test runs by cleaning some of them.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Thsi takes forevere to complete Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Also resolve conflicts This si for #1833 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

These are not needed and are too big. Remove a few other unused reqs. Also move dev dependencies off to the dev dir Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Only use the Scripts dir that is created by virtualenv Drop creating a junction as this is is error prone Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

This is self-contained adn does not need setuptools/wheel and pip It also works consistently with synced versions of its vendored deps And works consistently on Python 2 and 3. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

The latest url.py is written in C++ We do not need that speed but we need portability and pure python. The new urlpy is a fork of url.py v.2.0 that has been ported to Python 3 and works on 2 and 3. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

codecov · 2020-04-29T23:35:28Z

Codecov Report

Merging #1835 into develop will increase coverage by 0.02%.
The diff coverage is 87.50%.

@@             Coverage Diff             @@
##           develop    #1835      +/-   ##
===========================================
+ Coverage    79.54%   79.57%   +0.02%     
===========================================
  Files          131      131              
  Lines        16978    16941      -37     
===========================================
- Hits         13505    13480      -25     
+ Misses        3473     3461      -12

Impacted Files	Coverage Δ
src/cluecode/finder.py	`81.04% <87.50%> (-0.81%)`	⬇️
src/packagedcode/npm.py	`88.73% <0.00%> (-1.29%)`	⬇️
src/scancode/api.py	`96.17% <0.00%> (-1.28%)`	⬇️
src/packagedcode/pypi.py	`86.95% <0.00%> (-0.40%)`	⬇️
src/scancode/cli.py	`77.82% <0.00%> (-0.28%)`	⬇️
src/packagedcode/rubygems.py	`73.31% <0.00%> (+0.08%)`	⬆️
src/licensedcode/index.py	`73.64% <0.00%> (+0.22%)`	⬆️
src/licensedcode/query.py	`75.95% <0.00%> (+0.28%)`	⬆️
src/licensedcode/models.py	`82.75% <0.00%> (+0.36%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 694604c...694604c. Read the comment docs.

And improve testing and support of weird archives such as ar archives with illegal names. The enhancements to sevenzip extraction should allow things to work OK on the new macOS AFS filesystem. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

7zip can now also have ERROR-prefixed messages in stderr/stdout Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

we now can handle correcty the shortcomings of Apple AFS FS Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Somehow the file time is off by 2 hours and not GMT on Windows-only. And only when we list entries. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add CI debug for 7zip on macOS Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

But run them on Windows 32 and 64 bits Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

We no longer run coverage here for now. Also only run scancode. Do not check certain URLs. Set doc scripts as executable Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

They are the causes of too frequent random build failures as they make HTTP HEAD live calls and these can fail randomly. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne · 2020-07-06T11:21:11Z

At last this is passing on all OSes. Merging!

pombredanne added 3 commits November 18, 2019 20:00

Support Python 2.7+ and 3.6+

0f449b7

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not make doc deps part of requirements #1833

0b7f670

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not update homebrew on travis

fd36764

Thsi takes forevere to complete Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added the WIP label Jan 29, 2020

pombredanne added 25 commits March 12, 2020 17:39

Merge latest develop in 1833-install-pypi-py27

1385770

Also resolve conflicts This si for #1833 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge latest develop

ad235bb

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Test pip install on Python2 too

dcd7021

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Remove bundled docs dependent packages #1833

5802b2a

These are not needed and are too big. Remove a few other unused reqs. Also move dev dependencies off to the dev dir Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Streamline MANIFESTS for release

bb7792e

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add missing wheel for Python 3 #1833

f48e250

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Remove unsused package #1833

726bb40

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Correct ABOUT file for notice

492db7c

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add missing prebuilt maCOS deps #1833

60a2239

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Remove native URL lib #1833

b01b68d

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Bump attrs #1833

c414a00

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve declare license handling in Pypi #1833

519fed6

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Avoid using reserved names for variables

63adb1a

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Format code

08c98b9

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use correct ABOUT file

338826c

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use correct variable names

65899d7

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Reformat expected JSON file for whitespace

f0f2453

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update Pypi test expectatiosn #1833

7a3d704

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Include more than NOTICE in wheels

178c07e

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add Windows 64 as supported platform #616

3dc5863

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not use bin dir on Windows #616

f730584

Only use the Scripts dir that is created by virtualenv Drop creating a junction as this is is error prone Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Switch to use the virtualenv.pyz #1833

1d7f0ba

This is self-contained adn does not need setuptools/wheel and pip It also works consistently with synced versions of its vendored deps And works consistently on Python 2 and 3. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve doc wording

3f01487

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add libraries compiled for Windows 64 #616

fc029b0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

ncoghlan mentioned this pull request Jun 22, 2020

Upgrade scancode to Python 3.7 and 3.8 #1819

Closed

pombredanne added 5 commits June 25, 2020 19:55

Make sevenzip work file-by-file

798194c

And improve testing and support of weird archives such as ar archives with illegal names. The enhancements to sevenzip extraction should allow things to work OK on the new macOS AFS filesystem. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Ensure extraction still works on Python 2.7

5d166ab

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use plain path and support ERROR in stderr/stdout

6e68fce

7zip can now also have ERROR-prefixed messages in stderr/stdout Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve 7z error handling and windows tests

e2b02f0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not return warning message on macOS

6d0324b

we now can handle correcty the shortcomings of Apple AFS FS Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the 1833-install-pypi-py27 branch from cbe4966 to 6d0324b Compare June 27, 2020 19:47

pombredanne added 3 commits June 28, 2020 07:54

Ignore time for 7z entry listings on windows

3e7fdaf

Somehow the file time is off by 2 hours and not GMT on Windows-only. And only when we list entries. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add missing Windows 7z test files

0fa5af9

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Attempt to debug macOS sevenzip issues

db57ce5

Add CI debug for 7zip on macOS Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the 1833-install-pypi-py27 branch from dba5929 to db57ce5 Compare June 28, 2020 20:19

pombredanne added 13 commits June 28, 2020 23:09

Make more 7z tests pass on macOS

6be0524

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Updating expectations for macOS

1d8f07a

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update 7z expected JSON files formatting

8512ce6

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Make parsing 7z listing work on Linux

365cb33

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update Win 7z test expectations

84991f9

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use correct 7z test expectations on macOS

5b498ab

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Only run minimal smoke tests on Appveyor

34e5c13

But run them on Windows 32 and 64 bits Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use backslashes for Windows CI

a119268

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use spaces, not tabs

295ce0d

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Only run minimal tests on Travis

8695662

We no longer run coverage here for now. Also only run scancode. Do not check certain URLs. Set doc scripts as executable Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Remove verbose flag from tests

4e3dfc4

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use correct link to commit message documentation

065ca82

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add header comment to Azure pipeline

1bef9e2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the 1833-install-pypi-py27 branch from 1e9f402 to 1bef9e2 Compare July 6, 2020 06:24

pombredanne added 2 commits July 6, 2020 08:41

Do not check for broken links in .ABOUT files

e0f1b4f

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Remove link checks from Travis CI #2101

beab9a5

They are the causes of too frequent random build failures as they make HTTP HEAD live calls and these can fail randomly. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne merged commit f488b6c into develop Jul 6, 2020

pombredanne deleted the 1833-install-pypi-py27 branch July 6, 2020 11:21

Navigation Menu

Uh oh!

Uh oh!

Make PyPI installation work on Python 2 #1833 - #1835

Make PyPI installation work on Python 2 #1833 #1835
pombredanne merged 99 commits into
developfrom
1833-install-pypi-py27

pombredanne commented Nov 19, 2019 •

edited

Loading

Uh oh!

codecov Bot commented Apr 29, 2020 •

edited

Loading

Uh oh!

pombredanne commented Jul 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

pombredanne commented Nov 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pombredanne commented Jul 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pombredanne commented Nov 19, 2019 •

edited

Loading

codecov Bot commented Apr 29, 2020 •

edited

Loading