Skip to content

Wrong purl for PyPI packages in a conda environment? #462

New issue
New issue
Closed
Closed
Wrong purl for PyPI packages in a conda environment?#462
Labels
bugSomething isn't workingsource: conda

Description

@rolweber
rolweber
opened on Dec 12, 2022

When I generate an SBOM from "conda list --json" output, all components get a pkg:conda purl, even the non-conda packages installed from PyPI. If I understand the purl spec correctly, only the actual conda packages should get a pkg:conda purl.

Example:

    {
      "type": "library",
      "bom-ref": "6bca2ede-723c-4a33-aac8-9cb6e37f2ce2",
      "name": "ibm-watson-machine-learning",
      "version": "1.0.262",
      "purl": "pkg:conda/ibm-watson-machine-learning@1.0.262?build=pypi_0&channel=pypi&subdir=pypi",
      "externalReferences": [
        {
          "url": "https://conda.anaconda.org/pypi",
          "comment": "Distribution name ibm-watson-machine-learning-1.0.262-pypi_0",
          "type": "distribution"
        }
      ]
    }

When I generate the SBOM from "pip freeze" output, the same package gets a pkg:pypi purl.

    {
      "type": "library",
      "bom-ref": "0b38c1c3-7ad5-4f7d-b0db-1ce976a3eed9",
      "name": "ibm-watson-machine-learning",
      "version": "1.0.262",
      "purl": "pkg:pypi/ibm-watson-machine-learning@1.0.262"
    }

Of course, the conda-installed Python packages then also get a pkg:pypi purl, because pip doesn't know about conda.
However, conda does know about pip, and the non-conda packages are flagged as "pypi" in the output of "conda list --json".

I suggest to check for channel/subdir "pypi" when processing output from "conda list --json", and generate a pkg:pypi purl in that case.

Activity

  1. jkowalleck commented on Dec 12, 2022

    @jkowalleck
    jkowalleck
    on Dec 12, 2022
    Member

    Thanks for the report, @rolweber.

    It would be super helpful to have the exact output of conda list --json and pip freeze.
    It would also help to know the eexact command you are using for each cases - including all arguments/switches/parameters.

  2. jkowalleck
    added
    source: requirements
    source: conda
    on Dec 12, 2022

  3. rolweber commented on Dec 12, 2022

    @rolweber
    rolweber
    on Dec 12, 2022
    ContributorAuthor

    The environment I tested with has more than 380 packages. I'll create a smaller test environment tomorrow and will post the output then.

  4. rolweber commented on Dec 13, 2022

    @rolweber
    rolweber
    on Dec 13, 2022
    ContributorAuthor

    I'm using Linux x86. Creating a conda environment for testing...

    conda create -n Mixed python=3.10
conda run -n Mixed pip install seawater
conda env export -n Mixed

    Output of the last command:

    name: Mixed
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2022.10.11=h06a4308_0
  - certifi=2022.9.24=py310h06a4308_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.4.2=h6a678d5_6
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libuuid=1.41.5=h5eee18b_0
  - ncurses=6.3=h5eee18b_3
  - openssl=1.1.1s=h7f8727e_0
  - pip=22.3.1=py310h06a4308_0
  - python=3.10.8=h7a1cb2a_1
  - readline=8.2=h5eee18b_0
  - setuptools=65.5.0=py310h06a4308_0
  - sqlite=3.40.0=h5082296_0
  - tk=8.6.12=h1ccaba5_0
  - tzdata=2022g=h04d1e81_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.8=h5eee18b_0
  - zlib=1.2.13=h5eee18b_0
  - pip:
    - numpy==1.23.5
    - seawater==3.3.4

    conda list -n Mixed --json

    Details
    [
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "main",
    "channel": "pkgs/main",
    "dist_name": "_libgcc_mutex-0.1-main",
    "name": "_libgcc_mutex",
    "platform": "linux-64",
    "version": "0.1"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "1_gnu",
    "channel": "pkgs/main",
    "dist_name": "_openmp_mutex-5.1-1_gnu",
    "name": "_openmp_mutex",
    "platform": "linux-64",
    "version": "5.1"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h7b6447c_0",
    "channel": "pkgs/main",
    "dist_name": "bzip2-1.0.8-h7b6447c_0",
    "name": "bzip2",
    "platform": "linux-64",
    "version": "1.0.8"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h06a4308_0",
    "channel": "pkgs/main",
    "dist_name": "ca-certificates-2022.10.11-h06a4308_0",
    "name": "ca-certificates",
    "platform": "linux-64",
    "version": "2022.10.11"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "py310h06a4308_0",
    "channel": "pkgs/main",
    "dist_name": "certifi-2022.9.24-py310h06a4308_0",
    "name": "certifi",
    "platform": "linux-64",
    "version": "2022.9.24"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 1,
    "build_string": "h1181459_1",
    "channel": "pkgs/main",
    "dist_name": "ld_impl_linux-64-2.38-h1181459_1",
    "name": "ld_impl_linux-64",
    "platform": "linux-64",
    "version": "2.38"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 6,
    "build_string": "h6a678d5_6",
    "channel": "pkgs/main",
    "dist_name": "libffi-3.4.2-h6a678d5_6",
    "name": "libffi",
    "platform": "linux-64",
    "version": "3.4.2"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 1,
    "build_string": "h1234567_1",
    "channel": "pkgs/main",
    "dist_name": "libgcc-ng-11.2.0-h1234567_1",
    "name": "libgcc-ng",
    "platform": "linux-64",
    "version": "11.2.0"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 1,
    "build_string": "h1234567_1",
    "channel": "pkgs/main",
    "dist_name": "libgomp-11.2.0-h1234567_1",
    "name": "libgomp",
    "platform": "linux-64",
    "version": "11.2.0"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 1,
    "build_string": "h1234567_1",
    "channel": "pkgs/main",
    "dist_name": "libstdcxx-ng-11.2.0-h1234567_1",
    "name": "libstdcxx-ng",
    "platform": "linux-64",
    "version": "11.2.0"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h5eee18b_0",
    "channel": "pkgs/main",
    "dist_name": "libuuid-1.41.5-h5eee18b_0",
    "name": "libuuid",
    "platform": "linux-64",
    "version": "1.41.5"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 3,
    "build_string": "h5eee18b_3",
    "channel": "pkgs/main",
    "dist_name": "ncurses-6.3-h5eee18b_3",
    "name": "ncurses",
    "platform": "linux-64",
    "version": "6.3"
  },
  {
    "base_url": "https://conda.anaconda.org/pypi",
    "build_number": 0,
    "build_string": "pypi_0",
    "channel": "pypi",
    "dist_name": "numpy-1.23.5-pypi_0",
    "name": "numpy",
    "platform": "pypi",
    "version": "1.23.5"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h7f8727e_0",
    "channel": "pkgs/main",
    "dist_name": "openssl-1.1.1s-h7f8727e_0",
    "name": "openssl",
    "platform": "linux-64",
    "version": "1.1.1s"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "py310h06a4308_0",
    "channel": "pkgs/main",
    "dist_name": "pip-22.3.1-py310h06a4308_0",
    "name": "pip",
    "platform": "linux-64",
    "version": "22.3.1"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 1,
    "build_string": "h7a1cb2a_1",
    "channel": "pkgs/main",
    "dist_name": "python-3.10.8-h7a1cb2a_1",
    "name": "python",
    "platform": "linux-64",
    "version": "3.10.8"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h5eee18b_0",
    "channel": "pkgs/main",
    "dist_name": "readline-8.2-h5eee18b_0",
    "name": "readline",
    "platform": "linux-64",
    "version": "8.2"
  },
  {
    "base_url": "https://conda.anaconda.org/pypi",
    "build_number": 0,
    "build_string": "pypi_0",
    "channel": "pypi",
    "dist_name": "seawater-3.3.4-pypi_0",
    "name": "seawater",
    "platform": "pypi",
    "version": "3.3.4"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "py310h06a4308_0",
    "channel": "pkgs/main",
    "dist_name": "setuptools-65.5.0-py310h06a4308_0",
    "name": "setuptools",
    "platform": "linux-64",
    "version": "65.5.0"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h5082296_0",
    "channel": "pkgs/main",
    "dist_name": "sqlite-3.40.0-h5082296_0",
    "name": "sqlite",
    "platform": "linux-64",
    "version": "3.40.0"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h1ccaba5_0",
    "channel": "pkgs/main",
    "dist_name": "tk-8.6.12-h1ccaba5_0",
    "name": "tk",
    "platform": "linux-64",
    "version": "8.6.12"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h04d1e81_0",
    "channel": "pkgs/main",
    "dist_name": "tzdata-2022g-h04d1e81_0",
    "name": "tzdata",
    "platform": "noarch",
    "version": "2022g"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "pyhd3eb1b0_0",
    "channel": "pkgs/main",
    "dist_name": "wheel-0.37.1-pyhd3eb1b0_0",
    "name": "wheel",
    "platform": "noarch",
    "version": "0.37.1"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h5eee18b_0",
    "channel": "pkgs/main",
    "dist_name": "xz-5.2.8-h5eee18b_0",
    "name": "xz",
    "platform": "linux-64",
    "version": "5.2.8"
  },
  {
    "base_url": "https://repo.anaconda.com/pkgs/main",
    "build_number": 0,
    "build_string": "h5eee18b_0",
    "channel": "pkgs/main",
    "dist_name": "zlib-1.2.13-h5eee18b_0",
    "name": "zlib",
    "platform": "linux-64",
    "version": "1.2.13"
  }
]

    conda run -n Mixed pip freeze

    certifi @ file:///croot/certifi_1665076670883/work/certifi
numpy==1.23.5
seawater==3.3.4

    SBOM from conda list

    conda list -n Mixed --json | cyclonedx-bom -cj -i - -o - --format json | jq .

    The section for seawater is:

        {
      "type": "library",
      "bom-ref": "ddfcaeb2-ccb0-4a9c-b6d6-1ce7c26ea8da",
      "name": "seawater",
      "version": "3.3.4",
      "purl": "pkg:conda/seawater@3.3.4?build=pypi_0&channel=pypi&subdir=pypi",
      "externalReferences": [
        {
          "url": "https://conda.anaconda.org/pypi",
          "comment": "Distribution name seawater-3.3.4-pypi_0",
          "type": "distribution"
        }
      ]
    }

    SBOM from pip freeze

    conda run -n Mixed pip freeze >requirements.txt
cyclonedx-bom -r -o - --format json | jq .

    The section for seawater is:

        {
      "type": "library",
      "bom-ref": "c319ba1d-da80-42e1-94fb-25cb10591751",
      "name": "seawater",
      "version": "3.3.4",
      "purl": "pkg:pypi/seawater@3.3.4"
    }

  5. jkowalleck commented on Dec 15, 2022

    @jkowalleck
    jkowalleck
    on Dec 15, 2022
    Member

    fixed via #471

  6. jkowalleck
    closed this as completedon Dec 15, 2022
  7. jkowalleck
    added
    bugSomething isn't working
    and removed
    source: requirements
    on Dec 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsource: conda

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions