Repository navigation
Can not install custom python dependency from custom git url #555
Description
Description:
I am trying to install a python dependency from a custom git repo, with a specific commit and a subdirectory specified.
Steps to reproduce:
I have the following line in my requirements.txt:
langchain @ git+https://github.com/mrtj/langchain.git@7b52eff#subdirectory=libs/langchain
Observed result:
sam build fails with the error PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package
Expected result:
sam build works.
Note: pip install -r requirements.txt do correctly install the required version of the package.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
{
"version": "1.98.0",
"system": {
"python": "3.8.13",
"os": "macOS-12.5-x86_64-i386-64bit"
},
"additional_dependencies": {
"docker_engine": "20.10.23",
"aws_cdk": "Not available",
"terraform": "1.3.7"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}Add --debug flag to command you are running
Debug output:
2023-10-10 18:26:23,075 | Running PythonPipBuilder:ResolveDependencies
2023-10-10 18:26:23,104 | calling pip download -r /Users/janos/Projects/teamsystem_personalizer/personalizer/requirements.txt --dest
/var/folders/6r/wc15p6m13nl_nl_n_xfqpc5c0000gp/T/tmpvma9ri9e --exists-action i
2023-10-10 18:26:39,016 | Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for
help.\n\nError: No such option: -c\n"
2023-10-10 18:26:39,019 | Using fallback location for PKG-INFO file in package directory:
/var/folders/6r/wc15p6m13nl_nl_n_xfqpc5c0000gp/T/tmpve6hox12/langchain
2023-10-10 18:26:39,146 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
File "aws_lambda_builders/workflow.py", line 371, in run
File "aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
File "aws_lambda_builders/workflows/python_pip/packager.py", line 158, in build_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 259, in build_site_packages
File "aws_lambda_builders/workflows/python_pip/packager.py", line 283, in _download_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in _download_all_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in <setcomp>
File "aws_lambda_builders/workflows/python_pip/packager.py", line 515, in __init__
File "aws_lambda_builders/workflows/python_pip/packager.py", line 557, in _calculate_name_and_version
File "aws_lambda_builders/workflows/python_pip/packager.py", line 698, in get_package_name_and_version
File "aws_lambda_builders/workflows/python_pip/packager.py", line 610, in _get_pkg_info_filepath
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: langchain
2023-10-10 18:26:39,154 | Exception raised during the execution
Activity
lucashuy commentedon Oct 10, 2023
Hi, thanks for raising this issue. This is a result of how the package resolution logic works in Lambda Builders. Essentially, if the pip workflow finds a package that is not built already (the langchain fork that is provided), it'll try to resolve some basic info from that package by looking at the
setup.pyfile. Since this file is non-existent, it fails like you see here.While we look for a proper way forward, would it be possible to build this package locally and reference it inside of the
requirements.txtfile as a local dependency withfile://instead of pointing to the git repository? This could be done as a step prior to runningsam build, or as part of a Makefile build workflow.mrtj commentedon Oct 11, 2023
Hello, thank you for the workaround. Indeed compiling my version of langchain into a wheel and passing it as a local file pointer worked.
I am sure that you're aware of the fact that
setup.pyis deprecated since years and nowadays most projects usepyproject.tomlas package configuration structure. So looking for the existence ofsetup.pymight be error prone.Also, in langchain package the
pyproject.tomlis not placed in the root of the repo but instead in a subfolder pointed by the#subdirectoryURL tag as specified in the pip documentation. I do not know if this inferred with the bug somehow.lucashuy commentedon Oct 12, 2023
Thanks for confirming that building the project and then using the wheel file worked. As for moving away from setup.py, this would have to be something that would likely be taken in as a feature request since this is something that we don't currently check/handle.
Using the url fragments as you've done in the requirements file works fine however (though granted the project would probably have to have a
setup.pyfile as discovered)aws-lambda-builders requires setup.py when using git references (#7)
dtcaciuc-ggs commentedon Jun 28, 2024
I would also like to see
sam buildrecognizingpyproject.tomlfiles. We tried publishing wheels through build artifacts, however can't download them because the code is in a private repo and they cannot be accessed through SSH. We had to resort to hand-rolling PKG-INFO which is another thing to keep up to date and duplicates the info thatpyproject.tomlalready provides.