License ID mapping to Opensource.org

[pravinuttarwar]

The output of scancode gives good amount of details, but we want to show more data like keywords mentioned here https://github.com/OpenSourceOrg/api/blob/master/doc/endpoints.md, so we are thinking to somehow link scancode output license id to opensource api which also takes ID as input, any thoughts how we can combine these two and make it relevant?

[pombredanne]

@Mindbowser I could think of a script to update the data stored in Scancode YAML to add attributes and keywords from the OSI. And to sync things up like this is done with SPDX (see etc/synclic.py)

To do this there could be a couple steps:

1. establish a mapping between OSI and scancode license keys: fetch all OSi licenses, run scancode detection on them
2. store the osi "key" in a license and fetch OSI attributes using OSI. Store attributes in scancode licenses: for this you will need to update the licensedcode.models.License object as needed
3. return these as part of the scan

[pravinuttarwar]

Thanks @pombr, Why I need to run scancode detection on all OSI licenses? Also, where I need to store OSI key? Is there any yaml file present in scancode which I can update easily?

Can I use this plugin https://github.com/nexB/scancode-toolkit/wiki/License-Policy-Plugin? Where in my yml file I can add extra parameters for OSI?

[pombredanne]

@Mindbowser re:

Why I need to run scancode detection on all OSI licenses?

That would be to establish a mapping between the OSI files and their actual ScanCode license key

Also, where I need to store OSI key? Is there any yaml file present in scancode which I can update easily?

The best way way would be to store the in the YAML data file of a license the same way we store SPDX keys. These are in this directory: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses
For instance this is the SPDX license key attribute for a license:
https://github.com/nexB/scancode-toolkit/blob/58b313104e441c6f8ea98086bb5bf4060022bfb3/src/licensedcode/data/licenses/gpl-3.0.yml#L10

In the same way we could:

1. update the model to add an osi_license_key attribute https://github.com/nexB/scancode-toolkit/blob/58b313104e441c6f8ea98086bb5bf4060022bfb3/src/licensedcode/models.py#L135
2. update the YAML files for the licenses that have an OSI key

For that, we could:
A. use the OSI api to fetch all the licenses and their OSI key
B. match each to get their scancode license key (by calling licensedcode.index.match())
C. get a ScanCode License object from licensedcode.cache.get_licenses_db() (which is a mapping keyed by Scancode license key)
D. update this with the new osi_license_key attribute
E. and finally dump back the YAML https://github.com/nexB/scancode-toolkit/blob/58b313104e441c6f8ea98086bb5bf4060022bfb3/src/licensedcode/models.py#L238

[pombredanne]

@Mindbowser do you have any feedback on the above?

[pombredanne]

@ Mindbowser ping?

[gaupeng]

Can I have a go at this?
Also, can someone please explain how dump works? After I modify the License Model to include an OSI Key, and update the key, how exactly do I dump/modify the existing licenses?
@pombredanne @AyanSinhaMahapatra

[pombredanne]

@gaupeng re:

Can I have a go at this?

Yes and thank you!

Also, can someone please explain how dump works? After I modify the License Model to include an OSI Key, and update the key, how exactly do I dump/modify the existing licenses?

Say you add a new attribute called osi_license_key then you would do:

>>> from licensedcode.cache import get_licenses_db
>>> licenses = get_licenses_db()
>>> for lic in licenses.values():
...     lic.osi_license_key = 'Foo'
...     lic.dump()
... 

Or you edit by hand each file ... which is likely needed at some level?

[gaupeng]

@gaupeng re:
Say you add a new attribute called osi_license_key then you would do:

>>> from licensedcode.cache import get_licenses_db
>>> licenses = get_licenses_db()
>>> for lic in licenses.values():
...     lic.osi_license_key = 'Foo'
...     lic.dump()
... 

Sorry, I'm an absolute rookie, I figured out how to map the OSI Keys to the existing Scancode keys, however, after updating the License model (licensedcode/models.py) to have an attribute osi_license_key:

lic.osi_license_key = match

I get an invalid attribute error:

AttributeError: 'License' object has no attribute 'osi_license_key'

I'm guessing this is because the existing License objects do not have the osi_license_key attribute, and I'm trying to modify them directly. Any pointers please? @pombredanne

[pombredanne]

Can you please start a pull request? We can then review the code together.