The `--quiet` option doesn't suppress progress messages

[akaihola]

The scancode --help instructions advise to use the --quiet option to skip printing progress messages:

$ ./scancode --help
...
  --quiet                  Do not print progress messages.
...

However, that option has no effect. I still get the progress messages printed on stdout before the JSON output. This prevents me from running scancode as a subprocess and capturing its output as JSON:

$ ./scancode --quiet -l /tmp/LICENSE
Scanning files for: licenses with 1 process(es)...
Building license detection index...Done.
Scanning files...
Scanning done.
Scan statistics: 1 files scanned in 3s.
Scan options:    licenses with 1 process(es).
Scanning speed:  0.57 files per sec.
Scanning time:   1s.
Indexing time:   1s.
Saving results.
{... result JSON ...}

[pombredanne]

Thank you for reporting this! This is closely related to #388 from @sschuberth that is fixed but still needs work to implement the following scheme:

1. if no output file is specified, send the scan results to stdout
2. with no option: messages, progress bar and summary sent to stderr.
3. with the option --quiet no messages, no progress bar, no summary
4. with the option --verbose messages, per-file log and summary sent to stderr

The key would be to always send messages to stderr and the scan output to stdout
@akaihola would this work for you alright?

[akaihola]

@pombredanne, yes, that would be perfect.

Actually, before noticing the --quiet option, I was expecting to be able to read the JSON output from stdout and discard messages in stderr.

I'm currently working around this by doing the reverse: specify /dev/stderr as the output file, read JSON output from stderr and discard messages in stdout.

[pombredanne]

@akaihola No need to work around... I am working on a proper fix and will push it in a few

[pombredanne]

@akaihola @sschuberth The branch 388-417-consistent-stderr-stdout-outputs contains a proper implementation of the proposed update to outputs redirection for here and #388.
Feedback welcomed.

[pombredanne]

@akaihola Just curious you said:

This prevents me from running scancode as a subprocess and capturing its output as JSON

what is your use case there? I am always interested by new ways to use scancode!

[akaihola]

@pombredanne I tried out the 388-417-consistent-stderr-stdout-outputs branch. My observations:

• the --quiet option works as I expect it to
• messages are sent to stderr while JSON output goes to stdout
• however, the progress bar is still written to stdout:

$ ./scancode -l mypackage/LICENSE 2>/dev/null
  [####################################]  1                  
{... JSON output ...}

[akaihola]

@pombredanne Our use case is a Python script which

• accepts a list of Python package names (and optional versions) and/or paths to Python package source distribution tarballs/zipfiles
• downloads named packages from a package index server
• from existing and downloaded packages, extracts files whose names look like license files into a temporary directory
• reads license information from package metadata using pkginfo
• also identifies license(s) from extracted license files using Ninka and/or Scancode
• outputs a JSON license summary of the packages

I'm planning to suggest to my employer that we open source the script.

[pombredanne]

@akaihola Thanks for testing so quickly!

You wrote:

however, the progress bar is still written to stdout:

I will have to digg if this is an option with the click progressbar at all... if not we cqan consider using https://pypi.python.org/pypi/tqdm/ instead if it can echo to stderr (or patch and push a PR to click)

You use case is very interesting. A few notes:

1. afaik, there should not be a case where Ninka detects something that is not detected by ScanCode. If there is one, I would consider this as a bug to be fixed ASAP.

2. There is code in this branch https://github.com/nexB/scancode-toolkit/tree/253-python-packages that may not do exactly what you are looking for but is working towards it... And upcoming code to actually also fetch Pypi (and other package repos)

3. If you only look at files that look like a license, you may be missing out a few things IMHO (and FWIW there is simple code in scancode to do that here: https://github.com/nexB/scancode-toolkit/blob/538648e7bec139ee60e0ddf76481c0cda40eab84/src/licensedcode/legal.py but that has not yet been plugged in a scan proper

[pombredanne]

Actually I think I could simply override render_progress in https://github.com/pallets/click/blob/30fc626263db82e9f339ca8810d207ab089d5575/click/_termui_impl.py#L193 to ensure that the progress bar goes to stderr alright.