Regression in compileall ddir parameter when recursing

[hetman]

BPO	38715
Nosy	@serhiy-storchaka, @FFY00
PRs	bpo-38715: Fix compileall.compile_dir() ddir regression #17398

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-11-06.07:12:13.431>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'Regression in compileall ddir parameter when recursing'
updated_at = <Date 2020-05-03.17:15:40.122>
user = 'https://bugs.python.org/hetman'

bugs.python.org fields:

activity = <Date 2020-05-03.17:15:40.122>
actor = 'FFY00'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-11-06.07:12:13.431>
creator = 'hetman'
dependencies = []
files = []
hgrepos = []
issue_num = 38715
keywords = ['patch']
message_count = 3.0
messages = ['356102', '367984', '367986']
nosy_count = 3.0
nosy_names = ['serhiy.storchaka', 'hetman', 'FFY00']
pr_nums = ['17398']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38715'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[hetman]

There appears to have been a regression introduced in the compileall library back with Python 3.5 as a result of commit f1a8df0 .

The issues appears when compiling a hierarchy of directories with the 'ddir' parameter specified (either in the compile_dir() function or via the '-d' command line parameter).

Examining the code before the f1a8df0 commit suggests the original intent for the 'ddir' parameter, when recursing into subdirectories, was to alter the root of the path displayed in exception traces. That means that the hierarchical directory structure of the reported source path in each compiled file was maintained, and only the root directory being compiled was replaced with ddir.

More recently two new parameters, 'stripdir' and 'prependdir', have been added with commit 8e7bb99 which seem to largely reimplement the missing behaviour in a different way.

The current behaviour does not seem very useful or expected. I think it would be helpful to restore the original behaviour and would be happy to contribute a PR to do so. Additionally it may be worth consulting the submitter of bpo-38112 whether the alternative arguments are still necessary once the regression is fixed, though this is optional and not needed to fix the core issue.

Here's a simple script to demonstrate the above:

mkdir -p library/a/a

cat <<'EOF' > library/init.py
def go():
raise AssertionError('dummy')
EOF
cp library/init.py library/a/init.py
cp library/init.py library/a/a/init.py

python3 -m compileall -b -d /usr/lib library
rm library/**/*.py

python3 -c 'from library import go; go()'
python3 -c 'from library.a import go; go()'
python3 -c 'from library.a.a import go; go()'

And the resulting output which shows that each module is claiming to be raised from the same location:

Listing 'library'...
Compiling 'library/init.py'...
Listing 'library/a'...
Compiling 'library/a/init.py'...
Listing 'library/a/a'...
Compiling 'library/a/a/init.py'...

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/__init__.py", line 2, in go
AssertionError: dummy

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/__init__.py", line 2, in go
AssertionError: dummy

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/__init__.py", line 2, in go
AssertionError: dummy

[FFY00]

Can this be closed? I can't reproduce.

$ python -c 'from library.a.a import go; go()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/a/a/__init__.py", line 2, in go
    raise AssertionError('dummy')
AssertionError: dummy
$ python -c 'from library.a import go; go()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/a/__init__.py", line 2, in go
    raise AssertionError('dummy')
AssertionError: dummy
$ python -c 'from library import go; go()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/__init__.py", line 2, in go
    raise AssertionError('dummy')
AssertionError: dummy

[FFY00]

Nevermind! I ran it with my system python binary, which is patched to against this.

[hetmankp]

I believe this issue was identified again in #83950 and resolved with #18676, therefore this report can now be closed.