Server error 500 when clicking on some files

[ddmesh]

Hi,

when I start a scan via curl + REST API the scan runs successfully. When I then select some files in GUI
I get a server error 500 (http). Most other files can be opened, but few not.

I then run a bash on container "web" and could locate the file within the workspace/project directory and view it.
It has exactly the same and correct path as displayed in the tables via GUI (website).
It seems that URL (.../resources/123456/) does not resolve to the file's path correclty.
The path does not have any white spaces or risky characters.

Hope this can be solved easily.

Another questions: When there are some archives within the archive that I have uploaded, those are analysed. But I can also not display it. In workspace project directory there is no directory _extract and compliance alert says "missing". Any Idea how I can view those files too? I thought the default is that all archives within others are extracted.

Thanks a lot

[pombredanne]

@ddmesh Thanks fort the report!
Could you provide a small example of some open source archive to analyze that can reproduce your issue?
And provide the list of commands and possibly a screen shot?
Also how did you install SCIO and which version do you run?

[tdruez]

I thought the default is that all archives within others are extracted.

Which pipeline are you running? scan_codebase is not extracting recursively for example, see https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/scan_codebase.py#L53

compliance alert says "missing"

That's a different issue, what are the licenses on the resource for which the "missing" is returned? Do you have defined a policy for all of the licenses?

It has exactly the same and correct path as displayed in the tables via GUI (website).
It seems that URL (.../resources/123456/) does not resolve to the file's path correclty.

We need a bit more data to be able to debug this.

[ddmesh]

HI, Thanks for your fast answer.
I tried to provide an example where I could see this error. But I could not reproduce the first issue (error 500) when I exactly use the code from this repository. I tried to narrow down what differences cause my issue, but couldn't find the reason yet.
I use the same repository code, but added the policies.yaml, modified .gitignore and added private https root certificate for python to the docker file. Some of those changes
could be the reason. But the added certificates wasn't the reason, nor the policies-file. It somehow depends on
(re-)building scancode-io. Perhaps some caching plays its part too.

I use pipeline "scan_codebase" because I could't determine exactly the differences to the other ;-)
What pipeline should I use instead to have recursive extractions ?

The policies.yaml that I have used is from the tests directory. So I assume that the "MISSING" in the table means, that there is no entry for a specific license defined in policies.yaml?

I will do some more investigations and come back later.
Thanks a lot

[ddmesh]

Hi,
I have run more tests and figured out that when I place the policies.yml in "root" directory of scancode-io and
rebuild all with docker-compose build --force-rm --no-cache, I get a new column "Compliance Alert".

I can view all files where "Compliance Alert" are "empty" or contain "OK".
Files that show "MISSING" or "ERROR" can not be displayed.

I repeatedly placed or deleted the policies.yml file and run docker-compose build --force-rm --no-cache.
As result I have several projects created, some where I had the policies.yml and some that didn't had this file.
I only see the column "Compliance Alert" when the policies.yml was present when building scancode-io. But this applies to all projects no matter if those are scanned using policies.yml or not.
Here I would expect that each projects displays the column "Compliance Alert" with it was scanned with presence of policies.yml in depended of whether the current scancode-io was built with policies.yml file or not.

In case there is no column "Compliance Alert" I imagine there is one. I check the same files that had before "MISSING" or "ERROR" and where the scan was run when policies.yml file was used. Then I also get Error 500.
So the Error 500 depends on whether I use policies.yml or not and if the result in compliance-alert is "MISSING" or "ERROR".

For testing I uses the sources of scancode-io.
https://github.com/nexB/scancode.io/archive/516a12a9a6a21f23694adfaf7255bf3b7372fd05.zip

curl --show-error --progress-bar -X POST 'http://localhost/api/projects/' -F name='sc1' -F upload_file='@516a12a9a6a21f23694adfaf7255bf3b7372fd05.zip' -F pipeline='scan_codebase' -F execute_now=true

license_policies:
-   license_key: apache-2.0
    label: Approved License
    color_code: '#008000'
    compliance_alert: ''
-   license_key: mpl-2.0
    label: Restricted License
    color_code: '#ffcc33'
    compliance_alert: warning
-   license_key: gpl-3.0
    label: Prohibited License
    color_code: '#c83025'
    compliance_alert: error

To verify I selected the circle "programming language -> python" and checked the files that contained MISSING or ERROR in "compliances alert"

[pombredanne]

I use pipeline "scan_codebase" because I could't determine exactly the differences to the other ;-)
What pipeline should I use instead to have recursive extractions ?

We should consider these options:

1. be explicit that this pipeline extraction is "shallow" and have an alternative pipeline with "recursive" extraction
2. make this pipeline with "recursive" extraction by default
3. improve the way extraction is handled, such that we have progressive extraction as needed as opposed to blanket extraction

There are some things to consider when doing always recursive extraction with these existing issues in ScanCode toolkit:

• Transparent extraction of archives scancode-toolkit#14
• Quick Jar scan scancode-toolkit#218
• Report warning and not errors for certain extraction problems scancode-toolkit#1279

And some of the pending issues in https://github.com/nexB/extractcode/issues

I think this is worth making recursive the default (option 2) while we will work towards option 3.

@ddmesh What do you think?

[ddmesh]

Hi,
what about providing a configuration variable in file .env that controls extracting? so the user may choose as he needs?
Perhaps it is also possible to pass it as parameter when adding a pipeline via REST API or when using curl and form-based (-F) api?

[tdruez]

@ddmesh yes we'll provide a better way to control extracting. Until then, you can edit your local scan_codebase and set the extract_recursively argument to True.

[ddmesh]

I changed extract_recursively in my local copy of scancode-io.
Do you have already an idea about my findings in post #344 (comment) ?

[tdruez]

Do you have already an idea about my findings in post #344 (comment) ?

@ddmesh yes, I'll provide a fix for this, we should never return 500 in any cases. Thanks for the investigation.

[tdruez]

@ddmesh I've pushed a fix for the 500 error in main 73a0cd5
Could you give it a try and let me know if that solves it on your side?

[ddmesh]

@tdruez I have verified your fix and it is working now. The files can be viewed.

Since those changes I discovered that the scan ( same scancode-io archive) now produces two scan errors.

Model: DiscoverredPackage
Message: One or more of the required fields have no value: type, name, version
Detail: 
    ...
   type: pypi
  dependencies: [{'purl': 'pkg:pypi/importlib-metadata@3.10.0', 'scope': 'install', 'is_runtime': True, 'is_optional': False, 
  'is_resolved': True, 'requirement': '==3.10.0'}, {'purl': 'pkg:pypi/django@3.2.8', 'scope': 'install', 'is_runtime': True, 'is_optional': 
  False, 'is_resolved': True, 'requirement': '==3.2.8'}, {'purl': 'pkg:pypi/django-environ@0.7.0', 'scope': 'install', 'is_runtime': True, 
  'is_optional': False, 'is_resolved': True, 'requirement': '==0.7.0'},................
  primary_language: Python
All other fields are set to "None" or are empty.

[tdruez]

@ddmesh See the discussion at #338

[ddmesh]

Thanks, I will close this 500-Issue, as it is solved now. Thanks a lot for your effort. (Greetings from germany/dresden)