feat: parse `requirements.txt` with locally referenced packages

[madpah]

As per issue #284 raised by @Jonas-vdb.

Some development teams add local references to other packages in their requirements.txt files. This is currently causing an exception:

Traceback (most recent call last):
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 98, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

[madpah]

Root Cause: pkg_resources again does not support parsing requirements of this format (similar to requirements with hashes included) - see here.

Also - worth stating that given the example provided, we will not be able to determine the version for any locally referenced requirements through a requirements.txt method. Other methods such as EnvironmentParser would pick them up.

[jkowalleck]

@madpah i dont see an actual urge for this feature, and i dont see it as a bug.
relative paths are not to be expected to be output of pip freeze

but it is a convenient feature, no doubt.

[madpah]

I agree @jkowalleck - will remove the bug label.

[mostafa]

@madpah
I was thinking about making a PR to replace pkg_resources with requirements-parser because the latter can parse all sorts of requirements. WDYT?

[madpah]

Hey @mostafa - thanks for getting involved.

This was absolutely my plan. However, since this ticket was created, the logic/classes that use pkg_resources now live in an upstream project cyclonedx-python - so I'll first relocate this issue (back) there.

Would be great to get a PR from you to make this change, but worth nothing we have a large(ish) release due to land probably next week now, so you might want to wait until that is merged before you branch.

[madpah]

This issue was moved back to cyclonedx-python as since version 2.0.0 of cyclonedx-python-lib, Python specific parsers now reside in cyclonedx-python project.

[mostafa]

Perfect! Just give me a nod when you release next week. Alternatively, I can start working on the feature off the branch you want to merge and then rebase from the default branch after merging. Up to you! 🙂

[madpah]

Hey @mostafa - we've released cyclonedx-python 3.0.0 today. Would be awesome to get a PR to move to requirements-parser as you suggested.

Thanks!