Support multiple outputs from a single scan

[pombredanne]

It would be nice to have multiple outputs created at once. Today this requires re-running a scan as a scan has only a single output file.

[sschuberth]

This probably should be supported via something like --format json,spdx-tv.

[pombredanne]

@sschuberth How would you pass the path to the output file for each format then?
What about something like this?
--format-json foo/bat/scan.json --format-spdx-tv this/that/scan.spdx

[sschuberth]

Ah, sorry, I was too much thinking about how our own pre-processor takes command line arguments. We have a --format option that takes a comma-separated list of formats, and an --output-dir option that specifies the single output directory for all formats. The actual file base name is derived from the input directory / file.

If that's not flexible enough, I could also imagine --format to take a comma-separated list of file paths, and the output format is derived from the file extension, failing if it's not any of the known extensions.

I believe both suggestions above are more generic than having separate --format-XXX options for format XXX. Esp. if plugins may add new formats, is it even possible for plugins to add new command line options to the main program?

[pombredanne]

@sschuberth you wrote

We have a --format option that takes a comma-separated list of formats, and an --output-dir option that specifies the single output directory for all formats. The actual file base name is derived from the input directory / file.

I have thought of this approach, but that sounds to me as difficult to understand and removes control by the user of which files are created exactly. Passing a list of comma-separated file paths and inferring the format based on this feels also like a possible source of confusion..

I find it though to get the right CLI UI/UX here!
I wonder if there are other CLI tools that allow to specify multiple output and how they deal with it?

Or may be this is the wrong way to look at this entirely? What if instead we had ONLY a json output in ScanCode and a few additional small tools that could convert this output to another format (CSV, HTML, SPDX, etc)?

[sschuberth]

What if instead we had ONLY a json output in ScanCode and a few additional small tools that could convert this output to another format

That's what we do right now, after all, as we need your own "flavor" of SPDX-TV. While that approach certainly has its advantages, the one thing that I don't like about it is that you lose visibility over what output formats are supported. I simply like it when scancode --help tells me about that.

[pombredanne]

@sschuberth so you want scancode --help to tell you about all the available formats, as it should.
Right now, the help list only brief format "code names" right after the --format option.
If we were to use multiple options such as --json <path to output file>, -- spdx-tv <path to output file> we could still have a compact display in the help and these could still be grouped together.

Current way

  output:
    --strip-root           Strip the root directory segment of all paths. The
                           default is to always include the last directory
                           segment of the scanned path such that all paths have
                           a common root directory. This cannot be combined with
                           `--full-root` option.
    --full-root            Report full, absolute paths. The default is to always
                           include the last directory segment of the scanned
                           path such that all paths have a common root
                           directory. This cannot be combined with the `--strip-
                           root` option.
    -f, --format <format>  Set <output_file> format to one of: csv, html, html-
                           app, json, json-pp, jsonlines, spdx-rdf, spdx-tv or
                           use <format> as the path to a custom template file
                           [default: json]
    --verbose              Print verbose file-by-file progress messages.
    --quiet                Do not print summary or progress messages.

New way could be this:

  output:
  Available output formats   csv, html, html-app, json, json-pp, jsonlines,
                             spdx-rdf, spdx-tv or custom-format
                             [default: json]

    --json <file>            Save results to <file> in JSON compact format.
    --json-pp <file>         Save results to <file> in JSON pretty printed format.
    --jsonlines <file>       Save results to <file> in JSON lines format.
    --csv <file>             Save results to <file> in CSV format.
    --spdx-tv <file>         Save results to <file> in SPDX Tag-Value format.
                             This forces the --info scan option.
    --spdx-rdf <file>        Save results to <file> in SPDX RDF XML format.
                             This forces --info scan option.
    --html <file>            Save results to <file> in HTML format.
    --custom-template <file> Use the <file> as a custom format template [default:xyz]
    --custom  <file>         Save results to <file> in using a custom template format.
    --strip-root             Strip the root directory segment of all paths. The
                             default is to always include the last directory
                             segment of the scanned path such that all paths have
                             a common root directory. This cannot be combined with
                             `--full-root` option.
    --full-root              Report full, absolute paths. The default is to always
                             include the last directory segment of the scanned
                             path such that all paths have a common root
                             directory. This cannot be combined with the `--strip-
                             root` option.
    --verbose                Print verbose file-by-file progress messages.
    --quiet                  Do not print summary or progress messages.

You would still have a brief summary. Could something like this work for you?

[eclipsewebmaster]

Subscribed, that would be useful for us. Thank you

[pombredanne]

@sschuberth any feedback on my last comment?
@eclipsewebmaster any input on the proposed user experience?

[sschuberth]

Sorry for my late feedback, @pombredanne. I somewhat like your suggested "New way" command line help, however, I'd prefer to have all output format options prefixed with --output-*, like --output-json, --output-json-pp etc. to better explain what the format is about. In exchange, we probably can omit the explicit grouping of command line options.

Finally, I don't think there's much value in a separate brief summary.

[pombredanne]

@sschuberth Thank you! FWIW the grouping comes for free from the fact that these are output plugins. I like your explicit --output prefix anyway much better and no summary needed.

As a side note (as least on POSIX) we could also add shell completion (which is built in @mitsuhiko Click library)

[pombredanne]

Fixed in #885 !