Skip to content

Stop language SDK tasks at execution_timeout and report the timeout as the reason - #75446

Open
anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-75394-supervisor-timeout
Open

anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-75394-supervisor-timeout

Conversation

@anmolxlight

Copy link
Copy Markdown
Contributor

Part of #75394

When execution_timeout elapses, the supervisor (#74869) waited another killed_task_cleanup_time before signaling a language SDK task, and if the task then exited without reporting, the outcome was wrong:

Before After
Go / TypeScript / Java task gets SIGTERM execution_timeout + killed_task_cleanup_time execution_timeout
Stopped, exits without reporting, retries left stays running until the scheduler marks it killed externally up_for_retry
Stopped, exits without reporting, no retries failed, no reason failed
Stopped, swallows SIGTERM and exits 0 success failed
Reason none Task timed out after N seconds

The Python runner keeps its own SIGALRM timer for now, so for Python tasks the supervisor stays a backstop with the killed_task_cleanup_time grace period, as before. An outcome the task reports itself (for example RetryTask from a runtime that handles SIGTERM) still wins.

Not in this PR, since they need the decisions in the issue: removing the Python SIGALRM and asyncio.wait_for timers and choosing the signal that tells a Python task it timed out, and what to do with a task that reports success after its timeout.

…s the reason

The supervisor now sends SIGTERM to a language SDK task as soon as
execution_timeout elapses, instead of killed_task_cleanup_time later,
since these runtimes have no timer of their own. When the supervisor
stops a task that then exits without reporting an outcome, it reports
UP_FOR_RETRY or FAILED with the timeout as the reason, instead of
leaving the task instance running or marking a clean exit a success.

Part of apache#75394

Signed-off-by: anmolxlight <anmolxlight@gmail.com>
@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk labels Oct 23, 2026
Comment on lines +1837 to +1857
if self._execution_timed_out:
# The supervisor stopped the task, so report the timeout as the reason, including a retry, which
# would otherwise leave the task instance running until the scheduler finds it. A task that
# exits cleanly without reporting after being stopped has not succeeded either.
reason = f"Task timed out after {self._execution_timeout_seconds} seconds"
if self._should_retry:
self.client.task_instances.retry(
id=self.id,
end_date=datetime.now(tz=UTC),
rendered_map_index=self._rendered_map_index,
retry_reason=reason,
)
else:
self.client.task_instances.finish(
id=self.id,
state=TerminalStateNonSuccess.FAILED,
when=datetime.now(tz=UTC),
rendered_map_index=self._rendered_map_index,
retry_reason=reason,
)
return

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates _send_terminal_state_msg but drops its handling of CONFLICT, which sets SERVER_TERMINATED. If the server already moved the TI on, the raw retry() or finish() call raises.

Build a RetryTask(retry_reason) or TaskState(state=FAILED, retry_reason) instead, send it through _send_terminal_state_msg.

Comment on lines -2158 to +2194
def _arm_execution_timeout(self) -> None:
def _arm_execution_timeout(self, grace_period: float | None = None) -> None:
"""
Start the ``execution_timeout`` clock; the task is sent SIGTERM ``grace_period`` after it elapses.

``grace_period`` defaults to ``killed_task_cleanup_time``, the backstop for a Python task, which
raises ``AirflowTaskTimeout`` itself when the timeout elapses.
"""
if self._execution_timeout_seconds is None:
return
self._execution_timeout_enforce_at = (
time.monotonic() + self._execution_timeout_seconds + KILLED_TASK_CLEANUP_TIME
)
if grace_period is None:
grace_period = KILLED_TASK_CLEANUP_TIME
self._execution_timeout_enforce_at = time.monotonic() + self._execution_timeout_seconds + grace_period

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer polymorphism over a sentinel argument. Give ActivitySubprocess an overridable class attribute or property, such as _execution_timeout_grace_period, defaulting to KILLED_TASK_CLEANUP_TIME, and override it to 0 in _PopenActivitySubprocess.

Comment on lines 2030 to +2032
task a language SDK runs, it is armed once the runtime has been sent ``StartupDetails``. Either
way, the process is sent SIGTERM ``killed_task_cleanup_time`` after the timeout elapses.
way, the process is sent SIGTERM ``killed_task_cleanup_time`` after the timeout elapses, except a
language SDK task, which has no timer of its own and is sent SIGTERM when the timeout elapses.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Describe the behavior per subclass, or move it to where the override lives.

Comment on lines 2038 to 2042
self.process_log.error(
"Task did not stop before execution_timeout elapsed; terminating process",
timeout_seconds=self._execution_timeout_seconds,
grace_period_seconds=KILLED_TASK_CLEANUP_TIME,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a language SDK task, did not stop before execution_timeout elapsed with grace_period_seconds=KILLED_TASK_CLEANUP_TIME is misleading. There was no grace period, and that value is the SIGTERM to SIGKILL escalation delay. Rename the field or log the real grace period.

Comment on lines +1757 to +1765
@pytest.mark.parametrize(
("should_retry", "exit_code"),
[
pytest.param(True, -signal.SIGKILL, id="killed-with-retries"),
pytest.param(False, -signal.SIGKILL, id="killed-without-retries"),
pytest.param(False, 0, id="exited-cleanly-without-reporting"),
],
)
def test_execution_timeout_reported_as_reason(self, mocker, should_retry, exit_code):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gaps

  • Exit 0 without reporting while should_retry=True
  • An outcome reported after the timeout (e.g. RetryTask) still wins
  • The CONFLICT path
  • final_state for the exit-with-0 case

)

mock_monotonic.return_value += 35
# No grace period: the runtime has no timer of its own, so it is signaled at the deadline.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# No grace period: the runtime has no timer of its own, so it is signaled at the deadline.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants