Prevent Double Pipeline execution within a single Run

[tdruez]

In rare occurrences, a Pipeline may be executed again after completion in the same Task of the same Run.

For example:

2021-06-10 18:12:25.26 Pipeline [docker] starting
2021-06-10 18:12:25.27 Step [extract_images] starting
2021-06-10 18:12:28.79 Step [extract_images] completed in 3.53 seconds
2021-06-10 18:12:28.80 Step [extract_layers] starting
2021-06-10 18:12:34.56 Step [extract_layers] completed in 5.76 seconds
2021-06-10 18:12:34.57 Step [find_images_linux_distro] starting
2021-06-10 18:12:34.57 Step [find_images_linux_distro] completed in 0.00 seconds
2021-06-10 18:12:34.57 Step [collect_images_information] starting
2021-06-10 18:12:34.58 Step [collect_images_information] completed in 0.00 seconds
2021-06-10 18:12:34.58 Step [collect_and_create_codebase_resources] starting
2021-06-10 18:13:37.38 Step [collect_and_create_codebase_resources] completed in 62.80 seconds
2021-06-10 18:13:37.38 Step [collect_and_create_system_packages] starting
2021-06-10 18:18:39.73 Step [collect_and_create_system_packages] completed in 302.35 seconds
2021-06-10 18:18:39.73 Step [tag_uninteresting_codebase_resources] starting
2021-06-10 18:18:39.80 Step [tag_uninteresting_codebase_resources] completed in 0.06 seconds
2021-06-10 18:18:39.80 Step [tag_empty_files] starting
2021-06-10 18:18:39.81 Step [tag_empty_files] completed in 0.01 seconds
2021-06-10 18:18:39.81 Step [scan_for_application_packages] starting
2021-06-10 18:18:44.22 Step [scan_for_application_packages] completed in 4.41 seconds
2021-06-10 18:18:44.23 Step [scan_for_files] starting
2021-06-10 18:23:34.44 Step [scan_for_files] completed in 290.21 seconds
2021-06-10 18:23:34.44 Step [analyze_scanned_files] starting
2021-06-10 18:23:34.61 Step [analyze_scanned_files] completed in 0.17 seconds
2021-06-10 18:23:34.61 Step [tag_not_analyzed_codebase_resources] starting
2021-06-10 18:23:34.62 Step [tag_not_analyzed_codebase_resources] completed in 0.00 seconds
2021-06-10 18:23:34.62 Pipeline completed
2021-06-10 18:23:34.73 Pipeline [docker] starting
2021-06-10 18:23:34.74 Step [extract_images] starting
2021-06-10 18:23:38.17 Step [extract_images] completed in 3.44 seconds
2021-06-10 18:23:38.18 Step [extract_layers] starting
2021-06-10 18:23:44.15 Step [extract_layers] completed in 5.96 seconds
2021-06-10 18:23:44.15 Step [find_images_linux_distro] starting
2021-06-10 18:23:44.15 Step [find_images_linux_distro] completed in 0.00 seconds
2021-06-10 18:23:44.15 Step [collect_images_information] starting
2021-06-10 18:23:44.16 Step [collect_images_information] completed in 0.00 seconds
2021-06-10 18:23:44.16 Step [collect_and_create_codebase_resources] starting
2021-06-10 18:23:54.79 Step [collect_and_create_codebase_resources] completed in 10.63 seconds
2021-06-10 18:23:54.79 Step [collect_and_create_system_packages] starting
2021-06-10 18:28:46.65 Step [collect_and_create_system_packages] completed in 291.86 seconds
2021-06-10 18:28:46.66 Step [tag_uninteresting_codebase_resources] starting
2021-06-10 18:28:46.67 Step [tag_uninteresting_codebase_resources] completed in 0.01 seconds
2021-06-10 18:28:46.67 Step [tag_empty_files] starting
2021-06-10 18:28:46.68 Step [tag_empty_files] completed in 0.00 seconds
2021-06-10 18:28:46.68 Step [scan_for_application_packages] starting
2021-06-10 18:28:46.69 Step [scan_for_application_packages] completed in 0.01 seconds
2021-06-10 18:28:46.69 Step [scan_for_files] starting
2021-06-10 18:28:46.70 Step [scan_for_files] completed in 0.01 seconds
2021-06-10 18:28:46.70 Step [analyze_scanned_files] starting
2021-06-10 18:28:46.72 Step [analyze_scanned_files] completed in 0.02 seconds
2021-06-10 18:28:46.72 Step [tag_not_analyzed_codebase_resources] starting
2021-06-10 18:28:46.73 Step [tag_not_analyzed_codebase_resources] completed in 0.00 seconds
2021-06-10 18:28:46.73 Pipeline completed

Move investigation required.

from scanpipe.models import Run
Run.objects.filter(log__regex="(Pipeline completed).*(Pipeline completed)")

[tdruez]

Note that when reset and re-run, the pipeline may complete just fine. This means that the issue is probably not related to the content of the project.

[pombredanne]

See also #303

[tdruez]

@pombredanne As mentioned previously, resetting a project and running a pipeline from scratch usually works fine.
Looking at the logs, we can see that this issue occurred in the past using docker, root_filesystems, and scan_codebase pipelines on various projects.

It appears that this issue is:

• not related to a particular input
• not related to a particular pipeline

This is likely related to the task system but we didn't find a way to reproduce on demand as it seems to happen randomly so far.

[tdruez]

Fixed with the migration to RQ #333