back2source: Run pipeline for 100 .Net projects

[pombredanne]

.Net projects are mostly in C#, but also in C++. The special things are the WinPE binary format and the runtime.

[chinyeungli]

Created a list that can be used for testing
nuget_projects.xlsx

[tdruez]

@chinyeungli Could you update the list considering the following URL is not available

• https://github.com/ebekker/ACMESharp/releases/download/v0.8.1.0-EA/acmesharp-posh-all.0.8.1.0.nupkg#to could not be fetched.

[chinyeungli]

Updated the XLSX
nuget_projects-2025-01-16.xlsx

[tdruez]

Run on the 100 projects list provided at https://github.com/user-attachments/files/18427469/nuget_projects-2025-01-16.xlsx

1. Convert the nuget_projects-2025-01-16.xlsx into back2source_nuget_projects_list.csv

2. Batch create the projects

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/back2source_nuget_projects_list.csv \
    --pipeline map_deploy_to_develop \
    --label back2source-Nuget \
    --execute --async

3. Use the "Report" action on the filtered list of projects by "back2source-Nuget" label.

Results: back2source-report-Nuget.xlsx

[tdruez]

Re-run following the merge of #1674

LABEL="back2source-Nuget-v2"
INPUT="back2source_nuget_projects_list.csv"
PIPELINE_OPTIONS="Windows"

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/$INPUT \
    --pipeline map_deploy_to_develop:$PIPELINE_OPTIONS \
    --label $LABEL \
    --execute --async

Results: back2source-report-Nuget-v2-2025-06-20.xlsx

[pombredanne]

Results still need review

[AyanSinhaMahapatra]

See #1437 (comment) for issues added generally from all the d2d pipeline runs, many of which are also applicable for the nuget projects.

See also other issues specific to nuget:

• Get nuget symbols from XML file next to dll for matching binary-inspector#8
• Parse and get dependencies from .deps.json scancode-toolkit#4496
• Ignore specific paths/fles in .nupkg archives #1791
• Resources filter does not work for C# files #1792
• windows binaries from nuget has only CorDllMain symbol binary-inspector#7
• Improve binary symbol collection from .pdb files binary-inspector#9
• .dll.zsdiff and .exe.diff binary/archives not extracted binary-inspector#11

There are often also many extra .dll binaries included in nuget binary packages, see examples below:

In https://github.com/win-acme/win-acme/releases/download/v2.2.9.1701/win-acme.2.2.9.1701.nupkg:
GCC and other libraries related to C/C++ potentially:

to/tools/net7.0/any/libgcc_s_seh-1.dll
to/tools/net7.0/any/libgmp-10.dll 
to/tools/net7.0/any/libgnutls-30.dll
to/tools/net7.0/any/libhogweed-6.dll
to/tools/net7.0/any/libnettle-8.dll
to/tools/net7.0/any/libwinpthread-1.dll

Other kinds of .dll files:

*Microsoft.*.dll
to/tools/net7.0/any/Microsoft.Extensions.Configuration.Abstractions.dll

*System.*.dll
to/tools/net7.0/any/runtimes/win/lib/net7.0/System.Security.Cryptography.Pkcs.dll

Also many extra .dll in */runtimes/* which seems related to support of os/arch types:

• https://github.com/microsoft/garnet/releases/download/v1.0.48/garnet-server.1.0.48.nupkg
• https://github.com/git-ecosystem/git-credential-manager/releases/download/v2.6.0/git-credential-manager.2.6.0.nupkg