incorrect pipeline status when restarted the server

[chinyeungli]

I've created couple projects and launch a docker pipeline on each of them, but then the server is dead. Therefore, I restarted the server, and when I open the project page again, the "status" next to the pipelines is still showing "Running" which is incorrect.

[tdruez]

@AvishrantsSh I'd like to get your input on this one. Any ideas on how we could improve this?

To reproduce:

• Start the webserver: $make run
• Start any pipeline execution
• Kill the webserver process
• Restart the webserver and look at your pipeline state, still displayed as "running"

The pipeline will show as "running" forever while not running anymore. That status display is based on the database state of the Pipeline run, it was marked as "started" at the start of the pipeline execution, but since the worker process was "killed", it never got the chance to be marked as "ended/failed".

[AvishrantsSh]

@tdruez, I've looked into this issue and I believe there are a few workarounds:

• We can implement a custom signal for server shutdown and perform database cleansing at that time.
• When the server is started, we can look for entries in Run model that have no valid task_exitcode and perform cleansing accordingly. It can be implemented in wsgi.py using a filter like task_exitcode__isnull = True.

Personally, I feel that the second solution makes more sense as it is guaranteed that no celery job will be running at that time that might interfere with the cleansing. Besides, it is also simpler to implement than signals.

[AvishrantsSh]

I tried to implement the second solution and it seems to work perfectly.

[tdruez]

@AvishrantsSh thanks for your input. This approach works alright in local "runserver" mode.

But in the full stack mode (redis + celery worker + webserver + ALWAYS_EAGER = False) the "When the server is started" would not work since each service is run separately. The celery worker could still be working on a task while you restart the webserver for example.

[AvishrantsSh]

@tdruez As you rightly said, the approach that I mentioned should work in "offline" mode.
Now as for the complete solution, I have some doubts. First, can Celery update the db when django server is shut down? If it can then I don't think there will be any issue with this approach as we can set the cleansing to work only with local development.
However, if Celery can't update the record, then it is a different story.

[tdruez]

Celery update the db when django server is shut down?

Yes, the celery workers access the database independently from the django app.

Also, we can have multiple celery workers running, so doing the cleanup on starting a worker would not be good enough.

[AvishrantsSh]

@tdruez, can you provide me with instructions on how to test the full implementation. I tried doing it with a dockerized container of redis and using the command make worker, but for some reason, it always shows the result as a failure.

[tdruez]

@AvishrantsSh there's 2 ways to run the full ScanCode.io stack:

1. Running each service individually (dev mode, Linux and macOS only)

• Edit the .env file to disable the eager mode: CELERY_TASK_ALWAYS_EAGER=False
• Run a Redis server (https://redis.io/download): $ redis-server
• Run the Django app: $ make run
• Run the Celery worker:$ make worker

2. Docker (production mode)

• https://scancodeio.readthedocs.io/en/latest/docker-image.html#docker-image

[AvishrantsSh]

@tdruez, I've opened a pull request #171 for this issue. Please have a look and comment on any changes you'd like. I've used app.control.inspect() to check if any celery worker is online/working before the cleansing operation.

[tdruez]

@AvishrantsSh after looking deeper into this issue, the app.control.inspect() is not enough for a complete solution.

The celery inspect tool works at the workers level and knows only about the executing tasks and schedule ones (prefetched tasks). It does not provide an insight in the full queue of tasks. Those are waiting to be picked, by a worker, on the broker (redis) side.

The queued tasks can be gather using the redis api:

redis_connection = redis.StrictRedis(host="redis", port="6379", db=0)
tasks = redis_connection.lrange("redis://redis:6379/0", start=0, end=-1)

# or 
queue_name = "redis://redis:6379/0"
with celery_app.pool.acquire(block=True) as conn:
    tasks = conn.default_channel.client.lrange(queue_name, 0, -1)

I'm working on a solution that will combine those to recover all the tasks:

• Running: celery_app.control.inspect()
• Scheduled: celery_app.control.reserved() celery_app.control.scheduled()
• Queued: redis.lrange()