Integrate scancode license policy plugins

[pombredanne]

This https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/plugin_license_policy.py should be a great addition as a pipe in pipelines.
There are several interesting issues that need a solution:

1. running a standard plugin over a codebase stored in the DB
2. dealing with packages as the current plugin may only handle file/resource licenses
3. injecting the policy data which is a data file passed as a command line option in the toolkit

[tdruez]

While it's fairly easy to associate policy and license using a policies configuration file, the question is how do we want to present/compute all those policy values per object (resource or packages) on complex license expressions?

For example:
mit (policy=approved) AND gpl (policy=prohibited) AND mpl (policy=restricted)

What type of results would we want to present in the UI and in the output results for a codebase resource with this license expression?

[pombredanne]

While it's fairly easy to associate policy and license using a policies configuration file, the question is how do we want to present/compute all those policy values per object (resource or packages) on complex license expressions?

For example:
mit (policy=approved) AND gpl (policy=prohibited) AND mpl (policy=restricted)

What type of results would we want to present in the UI and in the output results for a codebase resource with this license expression?

We cannot guess:

1. I think we should obtain a precedence of policies (say here prohibited > restricted > approved) which means that here the overall policy would be prohibited. We likely would want to convert that to an intermediate expression, simplified and sorted by precedence as prohibited AND restricted AND approved and in these cases keep the left most (e.g. highest precedence policy)

2. There is the more complex cases of a choice: mit (policy=approved) OR gpl (policy=prohibited) and in this case there is a need to reason differently
   2.1. One way would be to apply apply a set of transforms to expression based on stated license choice preferences (here say mit over gpl or permissive over copyleft if at the category level) and therefore we would resolve this as policy=approved . This means having all license choices defined in additions to policies.
   2.2. Another way would be to obtain a preference for policies (say here approved > restricted > prohibited) and when we apply policies we could create an intermediate expression where we replace each license by its policy (say here we end up with approved OR prohibited and we use simplify the logical expression and finally apply the preferences above to each sub-expressions recursively and simplify once more at the end.
   2.3. Finally whether using 2.1 or 2.2, apply the precedence from 1. which would get us approved in both cases.

There is also something else to consider to make things much simpler: do NOT allow to define arbitrary policies. Instead define and hard code three policy values: OK/yes/approved, Needs review/maybe/restricted, NOT OK/no/prohibited ?
this would make the code much simpler and I am not sure that the loss of configurability there will be something we will ever miss (and if we miss it we know how to implement the more complex ways).
With this the precedence and preference do not have to be provided, because we know what they are always.