Implement DiscoveredDependency model

[JonoYang]

This PR adds the DiscoveredDependency model to scancode.io. New views, based off the existing DiscoveredPackage views, have been added for DiscoveredDependency.

This is the second PR split from #482 and is built on top of #485

[JonoYang]

@tdruez What would be a better way to write this migration query? https://github.com/nexB/scancode.io/blob/e2c9bcdae3530c379f17478f40303a94a9939659/scanpipe/migrations/0023_migrate_dependencies.py#L8

I am trying to populate the DiscoveredDependency table from the dependency data stored in the JSON fields on DiscoveredPackage before I remove the field.

[tdruez]

What would be a better way to write this migration query?

@JonoYang unfortunatly, I don't think there's an easy way here to use update() since the old dependencies field uses a complex data structure (JSON field as list of dict).
Since the amount of Packages is much less than Resources, I think we can go ahead with your approach.
Also, dependencies__isnull=False is not the proper lookup as it will return [] empty list value as well, and dependencies_data is not a field on the model, dependencies is.

I would refactor the code into something along:

def migrate_dependencies_to_discovereddependencies(apps, schema_editor):
    DiscoveredPackage = apps.get_model('scanpipe', 'DiscoveredPackage')
    DiscoveredDependency = apps.get_model('scanpipe', 'DiscoveredDependency')

    package_with_dependencies = DiscoveredPackage.objects.exclude(dependencies=[])
    
    for package in package_with_dependencies:
        for dependencies_data in package.dependencies:
            # Remove non-supported fields from the data dict
            dependencies_data.pop("extra_data", None)
            dependencies_data.pop("resolved_package", None)
            DiscoveredDependency.objects.create(project=package.project, **dependencies_data)

[tdruez]

@JonoYang What's the status on your side? Could you provide a list of what's left todo on this one?

1. I've made some changes to have a consistent "discovereddependencies" related name, this solves issues with the filter system, see 5d8cd43#diff-d71c8a8d98cec2c888bbf875d6a34a481041ba8a56656ec516aa53b0b7f6f194R1978

2. I've notice a UI issue for both "For package" and "Datafile resource" tab when there's no values to display:
   

3. Lastly, I've refactored the update_from_data method into a Mixin to remove the duplicated code, see dba4891#diff-d71c8a8d98cec2c888bbf875d6a34a481041ba8a56656ec516aa53b0b7f6f194R1013
   The only difference with the DiscoveredDependency.update_from_data is the following skip exception:
   field_name in PURL_FIELDS. Is it something we do not want to check on the dependency side?
   If that's the case, we can add a skip_purl_fields kwarg to the update_from_data method to make it usable for both models.
   We should also add a unit test for dependencies update similar to test_scanpipe_discovered_package_model_update_from_data

[JonoYang]

@tdruez

What's the status on your side? Could you provide a list of what's left todo on this one?

I think we're ready to go for now as the important stuff is here:

• model created
• pipelines updated to create DiscoveredDependencies and relate them to the Package and Resource they're from
• view updated to show DiscoveredDependency results and their relations

I was taking a look at #445 with this branch and I run into an issue with rubygem packages and their dependencies being reported multiple times. This is an issue with scancode-toolkit, and not scancode.io (aboutcode-org/scancode-toolkit#3072). I've also run into the license scan errors that you reported earlier.

I've made some changes to have a consistent "discovereddependencies" related name, this solves issues with the filter system, see 5d8cd43#diff-d71c8a8d98cec2c888bbf875d6a34a481041ba8a56656ec516aa53b0b7f6f194R1978

Thanks! I've updated the verbose_name and the verbose_plural_name and I was wondering why the related name didn't change.

The only difference with the DiscoveredDependency.update_from_data is the following skip exception:
field_name in PURL_FIELDS. Is it something we do not want to check on the dependency side?

I skipped this initially because we did not have purl fields in the DiscoveredDependency model. We can have this check again now that we're using the PURL field mixin in DiscoveredDependencies.