back2source: Run pipeline for Rust source and binaries

[pombredanne]

Most Rust projects are only provided as sources, and crates.io does not support binaries in general. So here we take released binaries. One specific thing for Rust is that these exists in multiple OS, with platform-independent data structures like cargo-auditable.

[tdruez]

@pombredanne We need the list of inputs to make progress on this one.

[AyanSinhaMahapatra]

@tdruez I have a list with me, let me upload a csv for this soon.

Maybe we should have the inputs (csvs with source/binary links), scan outputs (JSON) and the report csvs all at https://github.com/aboutcode-org/back2source-data in a structured format?

I can start a PR there

[tdruez]

@AyanSinhaMahapatra See #1476 (comment) for the supported CSV input example.

[AyanSinhaMahapatra]

@tdruez I've added aboutcode-org/back2source-data#6 with the input files, reports and scans for Rust. Ready for your review, is this an okay structure to store the data?

I'm adding the Scan data too, as this could be useful to look at improvements in various scans for improvements in SCIO d2d pipelines

[tdruez]

@AyanSinhaMahapatra Thanks for the listing at https://github.com/aboutcode-org/back2source-data/blob/fe50e25c82b70ceea9feba224bd50624ba77e47c/input-data/rust.csv
Although it only contains 25 projects we are generally running back2source on 100 projects.
Should we add more projects in this context or go ahead with 25 only?

[pombredanne]

@tdruez @AyanSinhaMahapatra 25 projects is a good enough start for now IMHO. We can expand this later, these project with binaries are harder to find for Rust and a few other ecosystem. We should in a near future run this at scale on many projects, like all rust crates!

[AyanSinhaMahapatra]

We can expand this later, these project with binaries are harder to find for Rust and a few other ecosystem

One approach to find these which I've used for rust and could also be helpful for other ecosystems is finding a github action which releases rust binaries and source, and then search github repos with that to get releases with rust binary and source archives.

For rust this was taiki-e/upload-rust-binary-action@v1 and tar: unix

[chinyeungli]

Create a project lists that use rust

back2source_rust_projects_list-2025-01-23.csv

[tdruez]

Run on the 100 projects list provided at https://github.com/user-attachments/files/18531274/back2source_rust_projects_list-2025-01-23.csv

1. Using back2source_rust_projects_list-2025-01-23.csv

2. Batch create the projects

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/back2source_rust_projects_list-2025-01-23.csv \
    --pipeline map_deploy_to_develop:Rust \
    --label back2source-Rust-v2 \
    --execute --async

3. Use the "Report" action on the filtered list of projects by "back2source-Rust-v2" label.

Results: back2source-report-Rust.xlsx

[chinyeungli]

Uploaded another project list with all the projects "release_download_url" point to x86 version

back2source_rust_projects_list-2025-03-14.csv

[tdruez]

Run on the 121 projects list provided at back2source_rust_projects_list-2025-03-14.csv

1. Using back2source_rust_projects_list-2025-03-14_v3.csv

2. Batch create the projects

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/back2source_rust_projects_list-2025-03-14_v3.csv \
    --pipeline map_deploy_to_develop:Rust \
    --label back2source-Rust-v3 \
    --execute --async

3. Use the "Report" action on the filtered list of projects by "back2source-Rust-v3" label. Select the "TODOs" sheet.

Results: back2source-report-Rust_v3.xlsx