back2source: Run pipeline for Python and C/C++ source and the python bytecode and native binaries

[pombredanne]

Once we run the d2d scans we are looking for problems, where a problem is defined as code that exists as source or binaries in the deployed wheels and that cannot be found in the sdist or the source repositories.

[tdruez]

@pombredanne We need the list of inputs to make progress on this one.

[chinyeungli]

uploaded the CSV (It contains projects with languages: python, c and cpp)

back2source_python_c_cpp_projects_list-2025-01-23.csv

[tdruez]

Run on the 100 projects list provided at https://github.com/user-attachments/files/18530686/back2source_python_c_cpp_projects_list-2025-01-23.csv

1. Using back2source_python_c_cpp_projects_list-2025-01-23.csv

2. Batch create the projects

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/back2source_python_c_cpp_projects_list-2025-01-23.csv \
    --pipeline map_deploy_to_develop \
    --label back2source-Python \
    --execute --async

3. Use the "Report" action on the filtered list of projects by "back2source-Python" label.

Results: back2source-report-Python.xlsx

[JonoYang]

I've rerun the projects using the updated d2d pipeline with cython matching using back2source-python-c-cpp-d2d.csv, and collated the results into a single xlsx file that is too large to upload here. At a quick glance, only sha1 matches and path matches were used. No binary matches were identified using the new python d2d step. Many to/ binaries are stripped and do not have any symbols or they are already built on the source (from/) side and matched using path/sha1 matching.

[JonoYang]

Using a more focused list of python packages with binary code (python-binary-packages.csv), I've run those projects using the d2d pipeline in scancode.io with the python and elf options enabled.

The projects had good mapping from source to binary, but a few had some issues:

• pyyaml 6.0.3: did not match to an unstripped binary, despite having pyx/pxd files
• patchelf-pypi 0.17.0.0: the binary in the to/ codebase is stripped, no matches can occur
• pycares 4.11.0: no cython files, source code that is compiled is inside huge comment blocks

The following are the d2d results:
results.zip