Repository navigation
Stop language SDK tasks at execution_timeout and report the timeout as the reason - #75446
anmolxlight wants to merge 1 commit into
Conversation
…s the reason The supervisor now sends SIGTERM to a language SDK task as soon as execution_timeout elapses, instead of killed_task_cleanup_time later, since these runtimes have no timer of their own. When the supervisor stops a task that then exits without reporting an outcome, it reports UP_FOR_RETRY or FAILED with the timeout as the reason, instead of leaving the task instance running or marking a clean exit a success. Part of apache#75394 Signed-off-by: anmolxlight <anmolxlight@gmail.com>
| if self._execution_timed_out: | ||
| # The supervisor stopped the task, so report the timeout as the reason, including a retry, which | ||
| # would otherwise leave the task instance running until the scheduler finds it. A task that | ||
| # exits cleanly without reporting after being stopped has not succeeded either. | ||
| reason = f"Task timed out after {self._execution_timeout_seconds} seconds" | ||
| if self._should_retry: | ||
| self.client.task_instances.retry( | ||
| id=self.id, | ||
| end_date=datetime.now(tz=UTC), | ||
| rendered_map_index=self._rendered_map_index, | ||
| retry_reason=reason, | ||
| ) | ||
| else: | ||
| self.client.task_instances.finish( | ||
| id=self.id, | ||
| state=TerminalStateNonSuccess.FAILED, | ||
| when=datetime.now(tz=UTC), | ||
| rendered_map_index=self._rendered_map_index, | ||
| retry_reason=reason, | ||
| ) | ||
| return |
There was a problem hiding this comment.
This duplicates _send_terminal_state_msg but drops its handling of CONFLICT, which sets SERVER_TERMINATED. If the server already moved the TI on, the raw retry() or finish() call raises.
Build a RetryTask(retry_reason) or TaskState(state=FAILED, retry_reason) instead, send it through _send_terminal_state_msg.
| def _arm_execution_timeout(self) -> None: | ||
| def _arm_execution_timeout(self, grace_period: float | None = None) -> None: | ||
| """ | ||
| Start the ``execution_timeout`` clock; the task is sent SIGTERM ``grace_period`` after it elapses. | ||
|
|
||
| ``grace_period`` defaults to ``killed_task_cleanup_time``, the backstop for a Python task, which | ||
| raises ``AirflowTaskTimeout`` itself when the timeout elapses. | ||
| """ | ||
| if self._execution_timeout_seconds is None: | ||
| return | ||
| self._execution_timeout_enforce_at = ( | ||
| time.monotonic() + self._execution_timeout_seconds + KILLED_TASK_CLEANUP_TIME | ||
| ) | ||
| if grace_period is None: | ||
| grace_period = KILLED_TASK_CLEANUP_TIME | ||
| self._execution_timeout_enforce_at = time.monotonic() + self._execution_timeout_seconds + grace_period |
There was a problem hiding this comment.
Prefer polymorphism over a sentinel argument. Give ActivitySubprocess an overridable class attribute or property, such as _execution_timeout_grace_period, defaulting to KILLED_TASK_CLEANUP_TIME, and override it to 0 in _PopenActivitySubprocess.
| task a language SDK runs, it is armed once the runtime has been sent ``StartupDetails``. Either | ||
| way, the process is sent SIGTERM ``killed_task_cleanup_time`` after the timeout elapses. | ||
| way, the process is sent SIGTERM ``killed_task_cleanup_time`` after the timeout elapses, except a | ||
| language SDK task, which has no timer of its own and is sent SIGTERM when the timeout elapses. |
There was a problem hiding this comment.
Describe the behavior per subclass, or move it to where the override lives.
| self.process_log.error( | ||
| "Task did not stop before execution_timeout elapsed; terminating process", | ||
| timeout_seconds=self._execution_timeout_seconds, | ||
| grace_period_seconds=KILLED_TASK_CLEANUP_TIME, | ||
| ) |
There was a problem hiding this comment.
For a language SDK task, did not stop before execution_timeout elapsed with grace_period_seconds=KILLED_TASK_CLEANUP_TIME is misleading. There was no grace period, and that value is the SIGTERM to SIGKILL escalation delay. Rename the field or log the real grace period.
| @pytest.mark.parametrize( | ||
| ("should_retry", "exit_code"), | ||
| [ | ||
| pytest.param(True, -signal.SIGKILL, id="killed-with-retries"), | ||
| pytest.param(False, -signal.SIGKILL, id="killed-without-retries"), | ||
| pytest.param(False, 0, id="exited-cleanly-without-reporting"), | ||
| ], | ||
| ) | ||
| def test_execution_timeout_reported_as_reason(self, mocker, should_retry, exit_code): |
There was a problem hiding this comment.
Gaps
- Exit 0 without reporting while
should_retry=True - An outcome reported after the timeout (e.g. RetryTask) still wins
- The CONFLICT path
final_statefor the exit-with-0 case
| ) | ||
|
|
||
| mock_monotonic.return_value += 35 | ||
| # No grace period: the runtime has no timer of its own, so it is signaled at the deadline. |
There was a problem hiding this comment.
| # No grace period: the runtime has no timer of its own, so it is signaled at the deadline. |
Part of #75394
When
execution_timeoutelapses, the supervisor (#74869) waited anotherkilled_task_cleanup_timebefore signaling a language SDK task, and if the task then exited without reporting, the outcome was wrong:execution_timeout + killed_task_cleanup_timeexecution_timeoutrunninguntil the scheduler marks it killed externallyup_for_retryfailed, no reasonfailedsuccessfailedTask timed out after N secondsThe Python runner keeps its own SIGALRM timer for now, so for Python tasks the supervisor stays a backstop with the
killed_task_cleanup_timegrace period, as before. An outcome the task reports itself (for exampleRetryTaskfrom a runtime that handles SIGTERM) still wins.Not in this PR, since they need the decisions in the issue: removing the Python SIGALRM and
asyncio.wait_fortimers and choosing the signal that tells a Python task it timed out, and what to do with a task that reports success after its timeout.