Version JSON output data format

[pombredanne]

As part of #2601 this is a first essential step before we start modifying more things to improve package reporting
This is also needed to support:

• Copyright and URLs data structure: Data structure inconsistency in results #2350
• Improve copryight data stucture consistency #2350 #2381
• Package license data structure: Improve tracing of license detection in package manifests #2389
• License data structure: In ouput include SPDX expressions, report matched rule by expression and do not report individual licenses #2278

[pombredanne]

Here is a first take on the policy there:

1. the version string is using this format scancode-toolkit-data-format-1.1 where the last two segments represent a semver-like version.
2. the first segment is the major version of the data format; it is incremented when there are attributes that are removed, renamed, changed or moved.
3. the second segment is the minor version of the data format; it is incremented when there are attributes that are only added.
4. we store the version string in our JSON output and display that also in the help, using a data_format_version attribute
5. this data format versioning is strictly for the JSON, YAML and JSON lines formats. It does not apply to CSV and any other formats. For these other formats there is no versioning and guaranteed format stability
6. for now, the format version is incremented by hand and only only increment per ScanCode tagged release is needed
7. We will document in the CHANGELOG the format changes in new format versions
8. In a given released code version, ScanCode TK may support only data format versions: the default, current version, and the next experimental version. We will update the CLI and functions to accept a new flag to select the next, experimental data format version (may be --next-data-format or --experimental-data-format or another flag name TBD)
9. By using --from-json xxx --json yyy we should be able to convert data from the current, default data format to the next, experimental format.
10. For any version we should provide a doc on the format Create "data dictionary" for all SCTK fields and schema #2008

[pombredanne]

@JonoYang @tdruez @AyanSinhaMahapatra @sschuberth @tsteenbe ping. Feedback welcomed. This is rather non-controversial.

[sschuberth]

it is incremented when there are attributes that are removed, renamed, changed or moved.

You probably should clarify that "moved" does not refer to changing the order at the same level, as that's not something what would break deserialization.

4. we store the version string in our JSON output and display that also in the help, using a data_format_version attribute

Maybe you should also make clear that the data_format_version attribute itself must never move, or that it always appears as the first attribute in the file, or something like that.

per ScanCode tagged release is needed

"if needed"

[pombredanne]

@mjherzog @DennisClark your comments are welcomed too.

[mjherzog]

This makes sense to me. It is a "nice" idea to have one versioning convention for a whole system, but we are all learning about the important differences in licensing between software and data. So this sounds like using the right tool for the job.

[JonoYang]

@pombredanne

Will the Codebase-Resource model schema from commoncode be versioned in the same way as the JSON output or is the output format independent of the Codebase-Resource model schema?

[pombredanne]

Will the Codebase-Resource model schema from commoncode be versioned in the same way as the JSON output or is the output format independent of the Codebase-Resource model schema?

@JonoYang that's a good point as these are tightly coupled . :|
This needs a bit of extra thinking.

[pombredanne]

@indirabhatt @maxhbr @soimkim ping too, FYI :)

[pombredanne]

Here is the updated documentation based on the feedback above (@sschuberth ❤️ ):

Output format version policy

We version the JSON output from ScanCode-Toolkit using this approach:

1. The version string is using this format scancode-toolkit-output-format-1.1 where the last segments after the dash represent a semver-like version of 1.1.

2. The first segment is the major version of the output format; it is incremented when attributes that are removed, renamed, changed or moved (but not reorder) in the JSON output. Reordering the attributes of a JSON object is not considered as a change and does not trigger a version change.

3. The second segment is the minor version of the output format; it is incremented when the changes are only for addition of attributes to the JSON output

4. We store the version string in the JSON output object as the first attribute and display that also in the help, using the new output_format_version attribute.

5. This output format versioning applies only to the JSON, pretty-printed JSON, YAML and JSON lines formats. It does not apply to CSV and any other formats. For these other formats there is no versioning and guaranteed format stability (or there some other rationale and convention for versioning like for SPDX)

6. For now, the output format version is incremented by hand and only incremented with a new ScanCode code tagged release if needed by output format changes.

7. We document in the CHANGELOG the output format changes in any new format version.

8. In a given released code version, ScanCode supports two output format versions: the default, current version, and the future version. The command line and core API functions will accept a new flag to select the future output format version (using --future-format option name).

9. When using --from-json xxx.json --json yyy.json --future-format we will able to convert data from a current, default JSON output format to the next, future JSON output format .

10. For any format version we will provide a documentation on the format and its updates using JSON examples and a comprehensive and updated data dictionary. See Create "data dictionary" for all SCTK fields and schema #2008 for details

[pombredanne]

After extensive review, supporting multiple versions of the output data format at once is an immense task! much simpler on paper than in practice... therefore I think will instead only track which version of data format is in a given SCTK version and we can commit to limit the number of major data format version changes to possibly no more than once a quarter.

The data format version and the documentation should be enough for users IMHO. The effort to have the current and future version would be similar to maintain two branches in the same codebase and make continuous forward port and back ports to each branch. This is too much work for too little benefits.