Add new "headers" top level attribute

[pombredanne]

• This is a new data structure as designed in RFC: Specify how header-level data are returned in ABCD aboutcode#7
• This would fix Provide header level details for a scan: Enhance scancode to include a log or history with useful statistics #211 and address the errors summary discussed in --only-findings should not omit scan_errors from JSON output #840
• For now, the old header-level data have been kept.
• The top level attribute is named headers as discussed RFC: Specify how header-level data are returned in ABCD aboutcode#7 ?

Some questions:

• old header-level attributes such as scancode_version : should we remove them entirely from the 3.0 output or deprecate them in doc and remove them only later?

@mjherzog @DennisClark @sschuberth @JonoYang @MaJuRG @mnonnenmacher feedback welcomed!

[pombredanne]

Note that this is the last feature-like ticket of the V3 milestone https://github.com/nexB/scancode-toolkit/milestone/8
After this only doc is left todo

[codecov]

Codecov Report

Merging #1285 into develop will decrease coverage by 0.36%.
The diff coverage is 96.22%.

@@             Coverage Diff             @@
##           develop    #1285      +/-   ##
===========================================
- Coverage    84.06%   83.69%   -0.37%     
===========================================
  Files          118      117       -1     
  Lines        12974    13305     +331     
===========================================
+ Hits         10906    11136     +230     
- Misses        2068     2169     +101

Impacted Files	Coverage Δ
src/plugincode/output.py	76.08% <ø> (ø)	⬆️
src/scancode/cli.py	76.57% <100%> (+0.66%)	⬆️
src/formattedcode/output_csv.py	84.72% <100%> (ø)	⬆️
src/formattedcode/output_jsonlines.py	100% <100%> (ø)	⬆️
src/formattedcode/output_html.py	91.41% <100%> (-0.06%)	⬇️
src/formattedcode/output_spdx.py	88.77% <100%> (+0.37%)	⬆️
src/formattedcode/output_json.py	77.35% <72.72%> (-0.42%)	⬇️
src/scancode/resource.py	84.28% <96.77%> (+0.31%)	⬆️
src/licensedcode/models.py	74.87% <0%> (-4.66%)	⬇️
src/licensedcode/index.py	80.09% <0%> (-1.83%)	⬇️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ddcc76...4083e72. Read the comment docs.

[sschuberth]

Please keep scancode_version (probably just renamed to "version" in new structure). We use that to determine whether scan results are directly comparable (different versions of ScanCode might detect different things).

And to be frank, I believe history_log to be a bad choice for a name. Neither "history" nor "log" (which also are somewhat redundant) capture the purpose properly, IMO. I'd probably generically call it info.

[pombredanne]

@sschuberth re

Please keep scancode_version (probably just renamed to "version" in new structure). We use that to determine whether scan results are directly comparable (different versions of ScanCode might detect different things).

This is kept and renamed to tool_version in the new structure such as here: https://github.com/nexB/scancode-toolkit/pull/1285/files#diff-a8e64273dc1cb3ca2a1ff8127a989c6cR136

Are you saying that you always need a top level scancode_version? or that you can get to the new version in history_log.0.tool_version alright?

And to be frank, I believe history_log to be a bad choice for a name. Neither "history" nor "log" (which also are somewhat redundant) capture the purpose properly, IMO. I'd probably generically call it info.

Yes! history_log is a bad choice!
... let's try something else: info is a candidate, this is a kind of "page info" that is offered in a browser, header is the name in the original ABC ticket.... let's find a few more candidate!

[pombredanne]

let's move the top level name discussion to aboutcode-org/aboutcode#7

[sschuberth]

Are you saying that you always need a top level scancode_version?

No, that's not a requirement for us, but we need to generally know from looking at the result file by which version of ScanCode it was created.

[pombredanne]

@sschuberth re

we need to generally know from looking at the result file by which version of ScanCode it was created.

that's going to be always available

[pombredanne]

@sschuberth I renamed this attribute to headers alright.

[pombredanne]

This looks all green now. Merging