Skip to content

Packagedcode to handle Python(requirements.txt) - #2078

Merged
pombredanne merged 6 commits into
aboutcode-org:developfrom
rpotter12:requirements-parser
Jul 13, 2020
Merged

pombredanne merged 6 commits into
aboutcode-org:developfrom
rpotter12:requirements-parser

Conversation

@rpotter12

Copy link
Copy Markdown
Contributor

Fixes #2037

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁

@rpotter12
rpotter12 force-pushed the requirements-parser branch 3 times, most recently from 588b2f7 to 63a9c30 Compare June 21, 2020 00:08
Comment thread src/packagedcode/pypi.py Outdated

@steven-esser steven-esser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few stylistic changes to be made, but more importantly, this code does not work correctly.

See comments below.

Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
@rpotter12
rpotter12 force-pushed the requirements-parser branch from 63a9c30 to a5e06c3 Compare June 23, 2020 01:04

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you checked dparse too? https://github.com/pyupio/dparse ?
It is using the "official" requirement handler from the "packaging" package https://github.com/pyupio/dparse/blob/master/dparse/parser.py#L15

@rpotter12
rpotter12 force-pushed the requirements-parser branch from a5e06c3 to f552a4f Compare June 27, 2020 03:37
@rpotter12

Copy link
Copy Markdown
Contributor Author

@MaJuRG @pombredanne
According to https://github.com/pyupio/dparse/blob/master/dparse/filetypes.py#L4 it parse files whose name is requirements.txt. I have done some minor changes and added test cases too :)

@rpotter12
rpotter12 requested a review from pombredanne June 28, 2020 22:58

@steven-esser steven-esser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty good. See one nitpick comment.

Comment thread tests/packagedcode/data/plugin/help.txt Outdated
@rpotter12
rpotter12 force-pushed the requirements-parser branch from f552a4f to f6bbaa1 Compare June 30, 2020 23:19
@rpotter12
rpotter12 requested a review from steven-esser June 30, 2020 23:19

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, we are almost there! 👍
See my comments inline.

Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
rpotter12 added 2 commits July 1, 2020 23:08
Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
@rpotter12
rpotter12 force-pushed the requirements-parser branch from f6bbaa1 to 83f29f9 Compare July 1, 2020 17:38
@rpotter12
rpotter12 requested a review from pombredanne July 1, 2020 17:41
Comment thread src/packagedcode/pypi.py
@rpotter12
rpotter12 requested a review from steven-esser July 2, 2020 21:39
@rpotter12

Copy link
Copy Markdown
Contributor Author

@MaJuRG @pombredanne If everything looks good please approve and merge this PR.:)

@steven-esser steven-esser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. @pombredanne Anything else to add here?

Comment thread src/packagedcode/pypi.py Outdated
Comment thread src/packagedcode/pypi.py
parsers = {
'setup.py': parse_setup_py,
'requirements.txt': parse_requirements_txt,
'requirements.in': parse_requirements_txt,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also note that for the future, I have submitted this PR upstream pyupio/dparse#48

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
@rpotter12
rpotter12 requested a review from pombredanne July 10, 2020 11:20

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks: Just a few last nits and this will be merged!

Comment thread tests/packagedcode/test_pypi.py
Comment thread tests/packagedcode/test_pypi.py Outdated
Comment thread src/packagedcode/pypi.py Outdated
Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
@rpotter12

Copy link
Copy Markdown
Contributor Author

@pombredanne Please review the PR again. :)

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you and I have a few last nits for you!

Comment thread src/packagedcode/pypi.py Outdated
return
file_name = fileutils.file_name(location)

file_name = get_dependency_type(file_name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be rather:
dependency_type = get_dependency_type(file_name)

Comment thread src/packagedcode/pypi.py Outdated

file_name = get_dependency_type(file_name)

if file_name not in (filetypes.requirements_txt,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if dependency_type not in (filetypes.requirements_txt,

Comment thread src/packagedcode/pypi.py Outdated
requirement = None
if specs:
requirement = str(specs)
if '==' in requirement:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about === and others quasi-equal in the latest specs?
Also since df_dependencies.specs is a list of Specifier packaging object https://github.com/pypa/packaging/blob/master/packaging/specifiers.py why not use that instead of taking a str() on it?
You could check if the .operator matches a list of equality operators AND that there is only one specifier (which is the only valid case for is_resolved )
See https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L173

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
@rpotter12
rpotter12 requested a review from pombredanne July 12, 2020 13:26
Comment thread src/packagedcode/pypi.py Outdated
name=df_dependency.name
).to_string()
if specs:
requirement = specs[0]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if there are more than one items in `specs?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you missed adding back something like:

requirement = str(df_dependency.specs)

?

Comment thread src/packagedcode/pypi.py Outdated
).to_string()
if specs:
requirement = specs[0]
if specs[0].operator == '==':

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if there are more than one items in specs?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is fixed now

Signed-off-by: rpotter12 <rohitpotter12@gmail.com>
@rpotter12
rpotter12 requested a review from pombredanne July 13, 2020 02:09

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now... Merging!
Thank you for your efforts there!
Note also this follow up ticket #2114

@pombredanne
pombredanne merged commit 3068680 into aboutcode-org:develop Jul 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parse requirements.txt files

5 participants