Description
get_python_version_os() in src/packagedcode/pypi.py silently drops the python_version environment marker for some comparison operators, because the list of accepted operators contains < twice and is missing >:
https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/packagedcode/pypi.py#L2281
python_version_operators = ['<', '>=', '==', '<=', '<']
< is duplicated where > was presumably intended, so > never matches. != and ~= are absent as well, though both are valid marker operators under PEP 508.
The value returned by this function becomes extra_data on the emitted DependentPackage:
extra_data = {}
if req.marker:
platform = get_python_version_os(req.marker)
if platform:
extra_data = platform
So a dependency declared as requests; python_version > "3.8" is reported with an empty extra_data, while requests; python_version >= "3.8" is reported correctly. The extracted marker data ends up inconsistent and incomplete depending only on which operator the package author happened to use.
How To Reproduce
Calling the current develop implementation over parsed requirements:
from packvers.requirements import Requirement
from packagedcode.pypi import get_python_version_os
for spec in [
'requests; python_version >= "3.8"',
'requests; python_version > "3.8"',
'requests; python_version != "3.8"',
'requests; python_version <= "3.8"',
'requests; python_version ~= "3.8"',
]:
print(spec, '->', get_python_version_os(Requirement(spec).marker))
Actual output:
requests; python_version >= "3.8" -> {'python_version': '>= 3.8'}
requests; python_version > "3.8" -> {} <-- dropped
requests; python_version != "3.8" -> {} <-- dropped
requests; python_version <= "3.8" -> {'python_version': '<= 3.8'}
requests; python_version ~= "3.8" -> {} <-- dropped
Expected: each of these should report the operator and version under extra_data["python_version"].
The same is visible end to end by scanning a requirements.txt that contains requests; python_version > "3.8" with:
scancode --package --json-pp - requirements.txt
and inspecting extra_data on the resulting dependency.
Suggested fix
Replace the duplicated < with the operators that are actually missing, covering the full set allowed for a marker in PEP 508:
python_version_operators = [
'<', '<=', '!=', '==', '>=', '>', '~=', '===',
]
System configuration
- What OS are you running on? Linux (x86_64)
- What version of scancode-toolkit was used? 33.0.0rc1,
develop at 5ebebf2
- What installation method was used to install/run scancode? source checkout
- Python version: 3.x
Description
get_python_version_os()insrc/packagedcode/pypi.pysilently drops thepython_versionenvironment marker for some comparison operators, because the list of accepted operators contains<twice and is missing>:https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/packagedcode/pypi.py#L2281
<is duplicated where>was presumably intended, so>never matches.!=and~=are absent as well, though both are valid marker operators under PEP 508.The value returned by this function becomes
extra_dataon the emittedDependentPackage:So a dependency declared as
requests; python_version > "3.8"is reported with an emptyextra_data, whilerequests; python_version >= "3.8"is reported correctly. The extracted marker data ends up inconsistent and incomplete depending only on which operator the package author happened to use.How To Reproduce
Calling the current
developimplementation over parsed requirements:Actual output:
Expected: each of these should report the operator and version under
extra_data["python_version"].The same is visible end to end by scanning a
requirements.txtthat containsrequests; python_version > "3.8"with:and inspecting
extra_dataon the resulting dependency.Suggested fix
Replace the duplicated
<with the operators that are actually missing, covering the full set allowed for a marker in PEP 508:System configuration
developat 5ebebf2