Error in `summary-by-facet` Plugin

[AyanSinhaMahapatra]

Description

Running the following command:
./scancode -clpieu --json-pp facet_summary.json samples --summary --facet dev=*.java --summary-by-facet

Generates the following Error

ERROR: failed to run post-scan plugin: summary-by-facet:
Traceback (most recent call last):
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/scancode/cli.py", line 1033, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 410, in process_codebase
    summarize_codebase_by_facet(codebase, **kwargs)
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 452, in summarize_codebase_by_facet
    [(key, sorted_counter(counter)) for key, counter in summary_counters])
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 448, in <genexpr>
    for key, values in summarizable_values_by_key.items()
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 305, in summarize_values
    return value_summarizers_by_attr[attribute](values)
KeyError: u'packages'

Scanning done.
Some files failed to scan properly:
ERROR: failed to run post-scan plugin: summary-by-facet:
Traceback (most recent call last):
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/scancode/cli.py", line 1033, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 410, in process_codebase
    summarize_codebase_by_facet(codebase, **kwargs)
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 452, in summarize_codebase_by_facet
    [(key, sorted_counter(counter)) for key, counter in summary_counters])
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 448, in <genexpr>
    for key, values in summarizable_values_by_key.items()
  File "/home/ayansm/Desktop/GSoD/scancode-toolkit-versions/scancode-toolkit-3.1.1/src/summarycode/summarizer.py", line 305, in summarize_values
    return value_summarizers_by_attr[attribute](values)
KeyError: u'packages'

System configuration

• What OS are you running on? Ubuntu 16.04 LTS
• What version of scancode-toolkit was used to generate the scan file? v3.1.1
• What installation method was used to install/run scancode? [source download]

[AyanSinhaMahapatra]

[Update]
Running ./scancode -clieu instead of ./scancode -clpieu doesn't give error.

[pombredanne]

thank you: good catch!
side note, always quote patterns in a facet to avoid shell glob expansion e.g. use dev="*.java" and not dev=*.java

[AyanSinhaMahapatra]

Noted. I will update docs accordingly.

[Iamshankhadeep]

and can I work on this too?

[pombredanne]

@Iamshankhadeep let's do one at a time if that works for you!

[inishchith]

@pombredanne Please let me know if I can work on this.

Thanks!

[pombredanne]

@inishchith please go for it: thanks! You may need to sync up with @JonoYang that is working in the same area too

[inishchith]

@pombredanne @JonoYang Any pointers on what work is done until now?
I could reproduce the error, but I'm finding it difficult to understand reason for the KeyError

[JonoYang]

@inishchith If you look in the error message, there is a filename and line number where the function of what is being called resides in. With this error, the problem is in line 305 of scancode-toolkit-3.1.1/src/summarycode/summarizer.py. The key error is occurring because we are not handling packages at all.

https://github.com/nexB/scancode-toolkit/blob/develop/src/summarycode/summarizer.py#L290

[inishchith]

@JonoYang Thanks for the response.

Yes, I get that.
The part which I didn't get ( because I have just started exploring the tool ) is what should be the result format for the package summary by facet? (an example would really help)

I've currently added a method ( which returns a counter object like: summarize_licenses ) which clears the error. I need an example result to know for sure what is expected. 😅

Thanks!

[JonoYang]

@inishchith I think it would make sense to make something like summarize_licenses (as you pointed to) that returns a mapping of package PURL and the number of instances of that package, i.e {package_purl: count of occurences}