Design and use a proper "universal" and unique package identifier

[pombredanne]

We need a proper package id that is universal and unique: the difficulty is that each package management technology uses more or less parts in an identifiers beyond basic name+version: Maven GAVs, RPM NEVRA, etc.
A simple solution is to have a single string ID with a prefix that describes what this is about and have a variable number of slash or colon-separated segments URN/URI-style such as used in:

• openshift and fabric8 analytics: Spike around mapping CVEs to individual packages of an ecosystem openshiftio/openshift.io#1052 (comment)
• the new grafeas https://github.com/Grafeas/Grafeas/blob/9d3c7b3a08c74dfa42e91c1ecd428e163f7bfe0a/README.md#resource-urls ( @R2wenD2 is this a convention that is already in use? do you have a longer list of examples? )
• CPEs, (more or less, with several caveats IMHO )
• DejaCode-style URNs

This should probably be defined in ABC Data and with aboutcode-org/aboutcode#6

[R2wenD2]

The naming conventions set forth by Grafeas are at least partially in use by Grafeas partners.
Unfortunately, there isn't currently a longer list.

[pombredanne]

@R2wenD2 Thank you for chiming in! Any pointer to some public or open source reference? or additional details you can share? Any documentation beyond your readme?

[pombredanne]

@mnonnenmacher @sschuberth ping, as this is a topic that is of interest you based on this discussion aboutcode-org/aboutcode#6 (comment)

[R2wenD2]

The best public pointer I can share are jFrog xray component identifiers It has small differences from Grafeas (docker and generic files), but the others match. Does that help?

We're using the identifiers suggested by the package manager where applicable and just prefixing with info about which package manager has specified them. I believe the open question here is about what we use to specify the package manager.

What is currently missing from the Grafeas list that you'd like to see?

[sschuberth]

@pombredanne FYI, the Grafeas spec is at https://github.com/Grafeas/Grafeas

[sschuberth]

@pombredanne Also, right now we use the tuple $packageManager:$namespace:$name:$version as the identifier, where $namespace is "The namespace of the package, for example the group id in Maven or the scope in NPM".

[pombredanne]

@sschuberth this approach works well for Maven and NPMs but may not be work for other package managers/formats/identifiers. Some remarks:

1. namespaces do not exist for Pypi and Rubygems for instance

2. extra identifying elements may be typically needed for Linux distro packages (e.g. rpm and deb) such as distro/os, arch (though release/version/epoch of rpms and similar for debs could be stuffed alright in a version)

3. some extra qualifiers may exists in a few common cases (java vs plain ruby for gems, various combos of os/arch for pypi, type/classifiers/packaging for maven)

So one thing is if we could make everything fit in a schema with a fixed number of segments OR have a variable number of segments depending on the package managers/format/repo technology.

I tend to think the later is more flexible and provide more resilience to changes for the future.

[R2wenD2]

I tend to agree with @pombredanne that the latter is more resilient and preferred.

[sschuberth]

1. namespaces do not exist for Pypi and Rubygems for instance

Correct. We simply use an empty thing in such cases.

2. and 3. could probably be captured by a generic optional extra qualifier field whose meaning depends on the package manager.

[pombredanne]

@R2wenD2 you wrote:

We're using the identifiers suggested by the package manager where applicable and just prefixing with info about which package manager has specified them. I believe the open question here is about what we use to specify the package manager.

Thank you for the xray pointer! Anyone there that you could ping?
I have minor consistencies comments e.g. for instance using gav for things that are from Maven repos and using pip which is the manager for Python pypi packages seem a tad weird as it is either uncommon (gav) or not clear if what is referenced is a packaging format as served by some repo technology (e.g. gav vs. maven) or an installation tool vs. a repo technology (e.g. pip vs. pypi) but this is a great base and it would be easy enough to create bidirectional mappings between similar conventions if needed.

What is currently missing from the Grafeas list that you'd like to see?

At least Rubygems, Composer, Golang and CPAN for a start and many more ;) I can suggest/contribute conventions back to Grafeas FWIW

[pombredanne]

@reiz @andrew out of curiosity, do you use any such thing in versioneye or libraries.io ?
@jpopelka your convention in openshift and fabric8 is "ecosystem:name:version" right? where the ecosystem is more or less the same as the grafeas "package manager" URL?

[jpopelka]

@jpopelka your convention in openshift and fabric8 is "ecosystem:name:version" right? where the ecosystem is more or less the same as the grafeas "package manager" URL?

Yes, our 'ecosystem' is basically a 'package manager'.

[andrew]

@pombredanne we don't have something similar to ecosystem/name/version but that doesn't quite enough to be truly unique it as it ignores architecture (rubygems for example can have different versions available for MRI/JRuby gems) and different registries (Maven central vs jfrog for example, but almost every package manager can point at different registries which may have the same name for a different package.

So really it needs to be something like registry-url/name/version/platform

For our purposes, especially considering Go (which has no registry), we've been treating the fully qualified url to the canonical package page as the unique identifier, although some package managers (like Bower) don't have individual urls for packages, so we just make one up from the registry domain.

[pombredanne]

@andrew Thanks! you wrote:

we've been treating the fully qualified url to the canonical package page as the unique identifier, although some package managers (like Bower) don't have individual urls for packages, so we just make one up from the registry domain

That's a clean and mostly universal approach too! But it does not always convey what a package "format" would be pointed to by a given URL unless this is for well known registries?

@R2wenD2 this brings up a possible ambiguities in the grafeas/xray approach:

• the identifier points unambiguously to a unique file or set of files for your htts://(docker) and file:// schemes
• OR to some package "name/version++" in other cases, and I assumed this would then be from the main public registry/repository for this package "format".

It does not include a notion of which repo/registry this packages lives in and furthermore each package name/version may have more than one "artifact" e.g. an sdist and many wheels for Python or an mri and jruby gem for Ruby, etc.

So in some cases you identify actual exact files or stack of files and in some other cases you identify some pointers to the primary public package repo for this format. This may be OK, but this may be also a source of confusion?