diff --git a/docs/source/dependencies-design.rst b/docs/source/dependencies-design.rst index 952f3817..113fa1c1 100644 --- a/docs/source/dependencies-design.rst +++ b/docs/source/dependencies-design.rst @@ -1,6 +1,6 @@ -========================================= - Resolving Python package dependencies -========================================= +==================================================== + Python package dependencies resolver design +==================================================== This is a design to create a new command line tool to resolve Python diff --git a/docs/source/index.rst b/docs/source/index.rst index 8a3c599d..c5ac78f0 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,11 +1,12 @@ -Welcome to nexb-skeleton's documentation! -========================================= +Welcome to python-inspector's documentation! +================================================ .. toctree:: :maxdepth: 2 :caption: Contents: dependencies-design + test-protocol Indices and tables ================== diff --git a/docs/source/test-protocol.rst b/docs/source/test-protocol.rst new file mode 100644 index 00000000..d5e918a8 --- /dev/null +++ b/docs/source/test-protocol.rst @@ -0,0 +1,308 @@ +============================================================== + Validation tests for Python package dependency resolution +============================================================== + +In order to validate that the python-inspector dependency resolution is correct +here is a procedure. + + +Requirements +*************** + +You need to: + +- have access to some Python codebase to use as a test bed. +- have a working installation of python-inspector +- have a working installation of scancode-toolkit v31.x (inlucing RCs) or higher + + +Test protocol +*************** + +Step 1: Collect development code details +-------------------------------------------- + +In this step, you will collect the list of requirements files that exist +in the codebase. You can use scancode:: + + scancode --package --info \ + --json-pp --processes 4 + +You then need to review the scanc results to assemble a list of pip +requirement files used in development. + +Or a simpler approach is to use the ``find`` command:: + + find -name "requirement*.txt" \ + > + +The output is a list of pip requirement files used in the development codebase. + + + +Step 2: Build your code +---------------------------- + +In this step, you need to run the build of your codebase and obtain the set +of deployed binaries. Building requires both dependency resolution and +installation of the resolved packages. The set of installed packages is used to +establish the ground truth and the expected results. + +As an output, keep the Python version that is used and the directory(ies) where +the deployed code is built named further down as . + + +Step 3: Collect built code details +--------------------------------------- + +In this step, you will extract the deployed code and run a scan to +collect existing packages as resolved during the original build. + +For this we will use extractcode and scancode with these commands:: + + extractcode --shallow + scancode --package --info \ + --json-pp --processes 4 + +You can adjust the number of processes up or down based on available CPU cores. + +The output is the file that contains all the detected +packages that were installed during the build. + + +Step 4: Resolve dependencies using development requirement files +-------------------------------------------------------------------- + +In this step, you will resolve the dependencies using python-inspector dad +command for each of the requirements files identified in Step 1 using the +Python version identified in Step 2. Run this command for each requirements +file, using each time a different output file name. We assume here Python +version 3.8 (note the absence of dot when passed as a command line option:: + + dad --python-version 38 --requirement \ + --json \ + --netrc + +The output is a list of JSON files with resolved packages for each of the +input requirements files. + + +Step 5: Collect expected resolved packages +---------------------------------------------- + +In this step, you need to collect the list of built packages "purl" found in +the top level "packages" attribute of the JSON scan output from Step 3. + +The output is a list of expected purls with a version. + + +Step 6: Collect actual resolved packages +---------------------------------------------- + +In this step, you need to collect the list of purls "package" attribute resolved +found in the top level "resolved_dependencies" attribute of the python-inspector +resolution output from Step 4. + +The output is a list of actual purls with a version. + + +Step 7: Compare expected with actual resolved packages +--------------------------------------------------------- + +In this step, you compare the output from Step 5 and Step 6 and validate the +correctness of python-inpector resolution. + +The output is a list of differences: + +- list 1: expected purls not present in the resolved purls. +- list 2: resolved purls not present in the expected purls. + + +(and if needed you can also collate the list of similar purls for reference). + + +If the build, build scan and resolution worked as expected, the list 1 and list 2 +should be empty. The differences need to be investigated. + +The possible causes could be: + +- extra actual resolved packages when some of the requirement files from Step 3 + also contains files for requirements that are not deployed (e.g. test, tools.) + +- missing actual resolved packages when the reference codebase from Step 1 uses + other build manifests beyond requirements.txt files. If this is the case, you + should carefully identify which other build manifests are used by reviewing + the development scan from Step 3. and report each manifest format as enhancement + request in the python-inspector issue tracker. + +- finally it can be a bug in python-inspector proper. Please report an issue + in the python-inspector issue tracker. Attach the list or JSON output(s) + from Step 1, Step 3 and Step 4. And the results of the review of Step 5, 6 + and 7. Alternatively you can share these files with python-inspector + maintainers if these are private. + + +End-to-end example +********************** + +Setup +------ + + +We use this repo https://github.com/tjcsl/ion as a sample codebase. +And the reference Python version is 3.8:: + + mkdir -p ~/tmp/pyinsp-example/ + cd ~/tmp/pyinsp-example/ + git clone https://github.com/tjcsl/ion + + +Another example could be https://github.com/digitalocean/sample-django + +We use the latest main branch from python-inspector and scancode-toolkit 31.0.0rc2 +installed on Linux with Python 3.8 using the release tarball from: +https://github.com/nexB/scancode-toolkit/releases/tag/v31.0.0rc2 + +ScanCode setup:: + + mkdir -p ~/tmp/pyinsp-example/tools + cd ~/tmp/pyinsp-example/tools + wget https://github.com/nexB/scancode-toolkit/releases/download/v31.0.0rc2/scancode-toolkit-31.0.0rc2_py38-linux.tar.xz + tar -xf scancode-toolkit-31.0.0rc2_py38-linux.tar.xz + cd scancode-toolkit-31.0.0rc2/ + ./scancode --help + +python-inspector setup:: + + cd ~/tmp/pyinsp-example/tools + git clone https://github.com/nexB/python-inspector + python3.8 -m venv venv + source venv/bin/activate + pip install --upgrade pip setuptools wheel + cd python-inspector + ./configure + +We will store all outputs in this directory:: + + mkdir -p ~/tmp/pyinsp-example/output + + +Step 1: Collect development code details +-------------------------------------------- + +We run a simple find:: + + find ~/tmp/pyinsp-example/ion \ + -name "requirement*.txt" > ~/tmp/pyinsp-example/output/requirements-file-paths.txt + +We find these two requirement files in ~/tmp/pyinsp-example/output/requirements-file-paths.txt:: + + ~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt + ~/tmp/pyinsp-example/ion/requirements.txt + + +Step 2: Build your code +---------------------------- + +We perform a simple "editable" build in place:: + + cd ~/tmp/pyinsp-example/codebase/ion + python3.8 -m venv venv + source venv/bin/activate + pip install --upgrade pip setuptools wheel + pip install --editable . + deactivate + + +Step 3: Collect built code details +--------------------------------------- + +We extract in place:: + + cd ~/tmp/pyinsp-example/tools/scancode-toolkit-31.0.0rc2/ + ./extractcode --shallow ~/tmp/pyinsp-example/codebase/ion + +And collect built details:: + + ./scancode --package --info ~/tmp/pyinsp-example/codebase/ion \ + --json-pp ~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json --processes 4 + +The output files is:: + + ~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json + + +Step 4: Resolve dependencies using development requirement files +-------------------------------------------------------------------- + + cd ~/tmp/pyinsp-example/tools/python-inspector + source venv/bin/activate + + dad --requirement ~/tmp/pyinsp-example/ion/docs/rtd-requirements.txt \ + --json ~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json + + dad --requirement ~/tmp/pyinsp-example/ion/requirements.txt \ + --json ~/tmp/pyinsp-example/output/resolved-requirements.txt.json + + deactivate + +The output files are:: + + ~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json + ~/tmp/pyinsp-example/output/resolved-requirements.txt.json + + +Step 5: Collect expected resolved packages +---------------------------------------------- + +Run this python script to generate text file with expected purls + +:: + + import json + with open("~/tmp/pyinsp-example/codebase/output/deployed-scan-output.json") as f: + scancode_data = json.load(f) + scancode_purls = [] + for package in scancode_data["packages"]: + if package["purl"] not in scancode_purls: + scancode_purls.append(package["purl"]) + scancode_purls = sorted(scancode_purls) + with open("~/tmp/pyinsp-example/codebase/output/scan.txt", "w") as f: + f.write("\n".join(scancode_purls)) + + +The output is a list of expected purls with a version. + + +Step 6: Collect actual resolved packages +---------------------------------------------- + +Run this python script to generate text file with actual purls + +:: + + import json + py_insp_purls = [] + for json_file in [ + "~/tmp/pyinsp-example/output/resolved-rtd-requirements.txt.json", + "~/tmp/pyinsp-example/output/resolved-requirements.txt.json", + ]: + with open(json_file) as f: + py_insp_data = json.load(f) + for package in py_insp_data["resolved_dependencies"]: + if package["package"] not in py_insp_purls: + py_insp_purls.append(package["package"]) + py_insp_purls = sorted(py_insp_purls) + with open("~/tmp/pyinsp-example/codebase/output/py-insp.txt", "w") as f: + f.write("\n".join(py_insp_purls)) + + +The output is a list of actual purls with a version. + + +Step 7: Compare expected with actual resolved packages +--------------------------------------------------------- + +We run a sdiff command:: + + sdiff ~/tmp/pyinsp-example/codebase/output/py-insp.txt ~/tmp/pyinsp-example/codebase/output/scan.txt diff --git a/tests/test_resolution.py b/tests/test_resolution.py index c8e8b94e..1832947f 100644 --- a/tests/test_resolution.py +++ b/tests/test_resolution.py @@ -82,11 +82,11 @@ def test_get_resolved_dependencies_with_flask_and_python_36(): assert as_list == [ "pkg:pypi/click@8.1.3", "pkg:pypi/flask@2.1.2", - "pkg:pypi/importlib-metadata@4.11.4", + "pkg:pypi/importlib-metadata@4.12.0", "pkg:pypi/itsdangerous@2.1.2", "pkg:pypi/jinja2@3.1.2", "pkg:pypi/markupsafe@2.0.1", - "pkg:pypi/typing-extensions@4.2.0", + "pkg:pypi/typing-extensions@4.3.0", "pkg:pypi/werkzeug@2.1.2", "pkg:pypi/zipp@3.8.0", ] @@ -107,7 +107,7 @@ def test_get_resolved_dependencies_with_tilde_requirement_using_json_api(): assert as_list == [ "pkg:pypi/click@8.1.3", "pkg:pypi/flask@2.1.2", - "pkg:pypi/importlib-metadata@4.11.4", + "pkg:pypi/importlib-metadata@4.12.0", "pkg:pypi/itsdangerous@2.1.2", "pkg:pypi/jinja2@3.1.2", "pkg:pypi/markupsafe@2.1.1",