Create purldb scan worker - #1078

Merged

tdruez merged 30 commits into

mainfrom

scio-purldb-scan-worker

Mar 27, 2024

pombredanne commented Feb 15, 2024 •

edited by mjherzog

Loading

Member

This PR is the ScanCode.io client side of the PurlDB scanning queue as designed in aboutcode-org/purldb#236

pombredanne marked this pull request as draft

February 15, 2024 10:02

pombredanne requested a review from JonoYang

February 15, 2024 10:02

pombredanne marked this pull request as ready for review

February 15, 2024 10:03

pombredanne mentioned this pull request

Expose new scan queue to many workers #49 aboutcode-org/purldb#290

Merged

tdruez commented Feb 19, 2024

Contributor

Tests are failing
TODOs to be addressed
Lack of unit tests for the new code

tdruez marked this pull request as draft

February 19, 2024 10:29

tdruez commented Feb 26, 2024

Contributor

@JonoYang What's the latest status on this one?

JonoYang force-pushed the scio-purldb-scan-worker branch from f36da81 to 068cfde Compare

February 26, 2024 21:57

JonoYang commented Feb 27, 2024

Member

I was working on updates to the purldb and API. I've updated my management command in this PR to make sure that basic functionality works (get download url, scan package, send results back), and that we can report scan project failures to purldb. I still need to create tests for management command functions.

pombredanne mentioned this pull request

Create client matching pipeline on scancode.io aboutcode-org/purldb#273

Closed

pombredanne commented Mar 12, 2024

Member Author

This needs rebasing/merging once #1077 is merged ...because of the renamings.

pombredanne mentioned this pull request

Improve indexing queue with multiple instances of ScanCode.io aboutcode-org/purldb#236

Closed

9 tasks

pombredanne assigned JonoYang

JonoYang force-pushed the scio-purldb-scan-worker branch from 56bf92c to ce66932 Compare

March 19, 2024 17:49

JonoYang requested a review from tdruez

March 20, 2024 02:03

tdruez requested changes

View reviewed changes

tdruez left a comment

Contributor

@JonoYang See my comments.
Also, should #1077 be merged first or after this one?

scanpipe/management/commands/__init__.py Outdated

scanpipe/management/commands/purldb-scan-queue-worker.py Outdated

scanpipe/management/commands/purldb-scan-queue-worker.py

+                          "0 means no limit. Used only for testing.",
+                      )
+                  def handle(self, *args, **options):

tdruez Mar 20, 2024

Contributor

This is too complex and needs to be refactored in smaller pieces

JonoYang Mar 27, 2024

Member

I've refactored the code in the handle() method. Is what I have still too complex?

scanpipe/pipes/purldb.py Outdated

scanpipe/pipes/purldb.py Outdated

scanpipe/pipes/purldb.py Outdated

scanpipe/pipes/purldb.py Outdated

scanpipe/tests/test_commands.py

Comment on lines +787 to +790

+                  class CreateProjectCommand(
+                      commands.CreateProjectCommandMixin, commands.AddInputCommandMixin, BaseCommand
+                  ):
+                      pass

tdruez Mar 20, 2024

Contributor

This setup seems too complex.

JonoYang Mar 21, 2024 •

edited

Loading

Member

Do you have any suggestions on how I should test the command methods? Should I have just called create-project using call_command instead of putting the project creation logic into a method?

tdruez Mar 25, 2024

Contributor

What about having the functions related to package creation outside the Command class? Those could be at the module level, what's your take?

JonoYang Mar 27, 2024

Member

@tdruez I am on board with that idea. What should be done about the messages that are printed using styling added by the django Command class? For instance, the calls to self.stdout.write() and self.stderr.write()? I think that's the main reason why I kept this code as Command mixins

tdruez reviewed

View reviewed changes

scanpipe/management/commands/__init__.py Outdated

JonoYang force-pushed the scio-purldb-scan-worker branch 2 times, most recently from 1360a05 to aa4427d Compare

March 23, 2024 00:36

JonoYang requested a review from tdruez

March 23, 2024 00:48

JonoYang commented Mar 23, 2024

Member

Also, should #1077 be merged first or after this one?

I think its alright to merge them in any order

JonoYang added 9 commits

March 26, 2024 14:24


          Create management command for purldb package scanning

a25c41c

Signed-off-by: Jono Yang <jyang@nexb.com>


          Fix bugs in package-scan-worker

91cb76c

Signed-off-by: Jono Yang <jyang@nexb.com>


          Send updates to purldb regarding scan status

c31e7e7

Signed-off-by: Jono Yang <jyang@nexb.com>


          Return scan project url when updating scan status on purldb

6bad6bc

Signed-off-by: Jono Yang <jyang@nexb.com>


          Get scananble_uri_uuid instead of package_uuid

bad9d62

     * Add sleep to main work loop

Signed-off-by: Jono Yang <jyang@nexb.com>


          Move project creation logic into its own function

    * Handle exceptions in package-scan-worker and send to purldb as errors

Signed-off-by: Jono Yang <jyang@nexb.com>


          Create project name from download url and uuid

6d65cb5

Signed-off-by: Jono Yang <jyang@nexb.com>


          Refactor poll_until_success

65ab289

Signed-off-by: Jono Yang <jyang@nexb.com>


          Add tests for new purldb functions

4b70406

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang added 19 commits

March 26, 2024 14:35


          Collect possible exceptions when getting next job

7a7fd04

Signed-off-by: Jono Yang <jyang@nexb.com>


          Get pipelines to run from purldb

05a684b

Signed-off-by: Jono Yang <jyang@nexb.com>


          Do not use sys.exit() in execute.py

Signed-off-by: Jono Yang <jyang@nexb.com>


          Add async arg to package-scan-worker.py

4bebc21

Signed-off-by: Jono Yang <jyang@nexb.com>


          Refactor commonly used command functions

1fffca4

Signed-off-by: Jono Yang <jyang@nexb.com>


          Create tests for Command mixins

c494f02

Signed-off-by: Jono Yang <jyang@nexb.com>


          Move get_next_job() to pipes/purldb.py

d500b5e

Signed-off-by: Jono Yang <jyang@nexb.com>


          Create test for package-scan-worker

085d034

Signed-off-by: Jono Yang <jyang@nexb.com>


          Create failure tests for package-scan-worker

38bf973

    * Remove `scanscan_project_url` from scanpipe.pipes.purldb.update_status()

Signed-off-by: Jono Yang <jyang@nexb.com>


          Save scannable_uri_uuid to project extra data

bc1fbed

Signed-off-by: Jono Yang <jyang@nexb.com>


          Bump matchcode-toolkit version

c6d50e6

Signed-off-by: Jono Yang <jyang@nexb.com>


          Send summary along with scan results to purldb

ccfc673

Signed-off-by: Jono Yang <jyang@nexb.com>


          Rename arguments for send_results_to_purldb

a7f48ee

    * Bump matchcode-toolkit version to 4.0.0

Signed-off-by: Jono Yang <jyang@nexb.com>


          Send project extra data to purldb with scan results

4401aea

Signed-off-by: Jono Yang <jyang@nexb.com>


          Print success message when results are sent

faa0957

Signed-off-by: Jono Yang <jyang@nexb.com>


          Update package-scan-worker tests

709d171

    * Sen both traceback and exception message to purldb

Signed-off-by: Jono Yang <jyang@nexb.com>


          Update CHANGELOG.rst

5c85dc6

    * Update management command test names

Signed-off-by: Jono Yang <jyang@nexb.com>


          Address review comments

1d472d5

    * Move lists of statuses in poll_until_success to their own variables
    * Remove unnecessary else statement
    * Use tuples as default values for `create_project`

Signed-off-by: Jono Yang <jyang@nexb.com>


          Refactor logic in purldb-scan-queue-worker

3c81a2d

    * Rename get_next_job to get_next_download_url
    * Update tests

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the scio-purldb-scan-worker branch from aa4427d to 76f8990 Compare

March 26, 2024 22:58


          Fix imports and tests after rebase

27ad732

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the scio-purldb-scan-worker branch from 76f8990 to 27ad732 Compare

March 27, 2024 01:27


          Move Command methods into functions

2fb7e12

Signed-off-by: Jono Yang <jyang@nexb.com>

tdruez marked this pull request as ready for review

March 27, 2024 11:17

tdruez merged commit 4f67f97 into main

tdruez deleted the scio-purldb-scan-worker branch

March 27, 2024 11:17

JonoYang mentioned this pull request

Scan queue updates aboutcode-org/purldb#285

Closed

7 tasks

tdruez mentioned this pull request

Unwanted log output while running tests #1142

Closed

pombredanne mentioned this pull request

ScanCode.io: Support multiple scan workers systems #1107

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet