Cancel queued and running pipeline task

[tdruez]

https://docs.celeryproject.org/en/latest/userguide/workers.html#revoke-revoking-tasks
https://docs.celeryproject.org/en/stable/reference/celery.contrib.abortable.html
https://www.ralphlepore.net/safely-cancel-your-celery-task/
https://stackoverflow.com/questions/30624290/celery-daemonic-processes-are-not-allowed-to-have-children

[tdruez]

See also #134

[tdruez]

https://docs.celeryproject.org/en/latest/userguide/workers.html#revoke-revoking-tasks

The app.control.revoke(task_id, terminate=True) would be the best approach but it is only supported by prefork pool while we are using the thread pool.

revoke: Revoking tasks
pool support: all, terminate only supported by prefork

[tdruez]

@pombredanne Note that we picked the "thread" pool instead of the default "prefork" one since we were encountering the following error when entering multiprocessing section of the code:
daemonic processes are not allowed to have children

This issue is related to a conflict between Celery that uses billard were the toolkit is using multiprocessing, if I remember correctly.

We may want to revisit this since the "thread" pool does not support several Celery features like terminate and timeouts.

[pombredanne]

@tdruez

We may want to revisit this since the "thread" pool does not support several Celery features like terminate and timeouts.

let's do it.
One thing we could do on the SCTK side is support receiving callables and pool-like objects as arguments such that it could use a SCIO-provided one when provided.

[tdruez]

@pombredanne The problem is not only on the TK side, we have entirely based the multiprocessing code in ScanCode.io on the concurrent.futures module.

[tdruez]

Current concurrency setup:

• Celery workers with pool=thread (based on concurrent.futures.ThreadPoolExecutor)
  -> Handles the parallelisation of the Pipelines executions
• ScanCode.io pipes based on concurrent.futures.ProcessPoolExecutor
  -> Handles the distribution of scanning processes over the multiple CPUs (multiprocessing)
• ScanCode scan_resource based on multiprocessing and _thread modules
  -> Handles the timeouts of scanning functions

---

We decided to use the thread pool for the Celery workers as the default prefork (based on billard does not work with the multiprocessing Python module that is used in ScanCode-toolkit and ScanCode.io

The issue: The thread pool does not support several Celery features like terminate and timeouts.

Possible solutions:

1. Keep everything as-is and implement custom methods for the terminate and timeouts on the TaskPool class.
   -> There's likely good reasons why Celery does not support this though...

2. Switch back to pool=prefork for Celery workers, rewrite the entire code for pipes based on ProcessPoolExecutor and tweak the toolkit to replace multiprocessing dependencies
   -> This will likely make the code very dependant on billard which is too specific to Celery

3. Simplify the whole concurrency setup to be only based on Celery, where everything down to single resource scanning would be an async task.
   -> We may need to deal with multiple queues to have separation between Pipeline execution and resource scanning. Also, cancelling a Pipeline execution for which thousands of resource scan tasks have been created may get even worst to handle.

4. Explore new queue library:

• https://python-rq.org/
• https://github.com/coleifer/huey/