Skip to content

Stop TypeScript SDK tasks at their execution_timeout - #76103

Open
shivaam wants to merge 2 commits into
apache:mainfrom
shivaam:ts-sdk-execution-timeout
Open

shivaam wants to merge 2 commits into
apache:mainfrom
shivaam:ts-sdk-execution-timeout

Conversation

@shivaam

@shivaam shivaam commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

A TypeScript task gets no signal at its execution_timeout. The runtime has no timer, so the task runs until the supervisor's backstop sends SIGTERM killed_task_cleanup_time (60 s) later. If the handler then returns normally, the task is recorded as success and its retries are skipped.

The TypeScript runtime now handles the timeout itself, as the Python task runner does with SIGALRM:

  • It reads execution_timeout from the run context. Unset or 0 means no timeout, as for a Python task. A negative value has already passed, so the handler does not run.
  • Right before the handler (or operator) runs, it sends SetExecutionTimeout with that timeout, so the supervisor's backstop uses the same value and counts from the same moment.
  • At the timeout it aborts getContext().signal with a new exported TaskTimeoutError as its reason. It then waits up to 30 s (ABORT_GRACE_PERIOD_MS) for the handler to clean up, or less if the supervisor's SIGTERM comes first. Then it fails the task with Task timed out after N seconds, as RetryTask if it has retries left. What the handler returns after the timeout is ignored, so a timed-out task can no longer succeed.
  • A handler stuck in a synchronous loop cannot see the signal; the supervisor's backstop still stops it, unchanged.
With execution_timeout=2, retries=1 Before After
Handler passes the signal to its work stopped at 62 s; reason "The operation was aborted" stopped at 2 s (+ its cleanup); reason "Task timed out after 2 seconds"
Handler returns after being aborted success at 62 s, no retry up_for_retry at 2 s
Handler ignores the signal killed at 92 s, no reason timeout reported at 32 s with its reason
Synchronous loop SIGKILL by the supervisor unchanged

Those rows are from running a bundle built from this branch, and one built from main, under the real Python supervisor (_PopenActivitySubprocess, real sockets and signals; only the Execution API was mocked).

Not in this PR, as both need the coordinator to pass a value to the runtime: falling back to the worker's default_task_execution_timeout when the run context has no timeout, and waiting killed_task_cleanup_time instead of a fixed 30 s after the timeout.

Validation:

  • Rebased on Send the Python task's execution_timeout to the supervisor #75387. Full TS suite: 977 tests passed, including 9 new unit cases and 11 new runtime cases against the simulated supervisor.
  • Typecheck, ESLint, Prettier, build and verify:package passed; prek hooks passed on commit.
  • 14 cases (7 handlers × branch/main) under the real Python supervisor; after the rebase, the branch cases rerun against the merged supervisor give the same results, and it accepts SetExecutionTimeout.
  • Full Airflow run on the rebased branch (api-server, scheduler with LocalExecutor, Dag processor, NodeCoordinator, SQLite): one TypeScript Dag packed with airflow-ts-pack against this branch and against main. On main, the task that returns after the abort ends success with its retry skipped; with this change both tries fail with "Task timed out after 3 seconds". A handler that passes the signal stops at the timeout instead of at the supervisor's SIGTERM, and a synchronous loop is still killed by the supervisor.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

@ChickenBenny

ChickenBenny commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Hi @shivaam, thanks for doing this!

Some context on where #75387 landed, from @jason810496's reviews (#75387 (review) and #75387 (review)):

  • The API server returns execution_timeout in ti_context only on the is_stub path, since the runtime can't resolve the value set on a Python @task.stub itself. Unset means the task didn't set one, 0 means no timeout.
  • The coordinator passes the worker's default to the runtime as AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT.
  • Each SDK picks the value (ti_context, else the worker default) and sends SetExecutionTimeout(timeout_seconds) right before running, the same as a Python task.

This PR also covers what @Andrushika raised in #74869 (#74869 (comment)): language SDK tasks had nothing telling them about the timeout, so they only got SIGTERM at execution_timeout + killed_task_cleanup_time. Nice to see that fixed.

Compared with the above, a few things:

  • I think it'd be easier to land this after Send the Python task's execution_timeout to the supervisor #75387, otherwise a few things won't match. On main the supervisor only takes StartExecutionTimeout (after Send the Python task's execution_timeout to the supervisor #75387 it's SetExecutionTimeout with the timeout the runtime picked), and the server still clamps a negative timeout to 0, so on main a 0 actually means "already elapsed", not "no timeout". This PR already follows the Send the Python task's execution_timeout to the supervisor #75387 meaning, so rebasing onto it once it's merged should be mostly the message change.
  • executionTimeoutOf treats unset as no timeout. With the worker default, unset should fall back to AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT instead. It doesn't matter today because stubs still carry the Dag processor's default, but it will once follow-up 1 lands. Do you want to add it here or in a separate PR?
  • The 30s wait after the timeout is fixed, while the supervisor uses killed_task_cleanup_time. Is that on purpose?

Otherwise looks good to me 👍

Related: #74716, #74717, #74869, #75387


Drafted-by: Claude Code (Opus 5.5); reviewed by @ChickenBenny before posting

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should follow the direction that @ChickenBenny pointed out, thanks.

@shivaam

shivaam commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ChickenBenny and @jason810496, agreed. I'll rebase once #75387 merges and send SetExecutionTimeout(timeout_seconds) right before the task's work starts.

On the 30 s wait: agreed, we can use killed_task_cleanup_time once the coordinator passes it to the runtime. Here is how the two clocks line up for a task with execution_timeout = 39 s and the default killed_task_cleanup_time = 60 s:

              T           T+39                        T+99                   T+159
              |-----------|---------------------------|-----------------------|
TS runtime    task runs   timeout: abort signal,      report "Task timed out
                          wait for cleanup            after 39 seconds", exit

supervisor    counting ...                            SIGTERM              SIGKILL
                                                      (each only if the process is still alive)
  • At the timeout the runtime aborts the task's signal with TaskTimeoutError and waits for it to clean up.
  • With the config value, that wait ends at T+99, the same moment the supervisor sends SIGTERM. Whichever comes first, the runtime reports the timeout and exits.
  • SIGKILL 60 s later only matters for a process that cannot stop itself, such as a synchronous loop.
  • Today the runtime waits a fixed 30 s (the same wait it gives after SIGTERM), so it reports at T+69. If killed_task_cleanup_time is under 30 s, the supervisor's SIGTERM arrives first and the runtime stops waiting then.

Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting

A TypeScript task got no signal at its execution_timeout: the runtime had
no timer, so the task ran on until the supervisor's backstop sent SIGTERM
killed_task_cleanup_time later, and a handler that then returned normally
was recorded as a success, skipping its retries.

The runtime now handles the timeout itself, as the Python task runner does:

- It reads execution_timeout from the run context (unset or 0: none;
  negative: already passed, so the handler does not run).
- Right before the handler or operator runs, it sends
  SetExecutionTimeout with that timeout, so the supervisor's backstop
  uses the same value and counts from the same moment.
- At the timeout it aborts getContext().signal with a new exported
  TaskTimeoutError, waits up to 30 seconds for the handler to clean up
  (less if the supervisor's SIGTERM comes first), then fails the task
  with "Task timed out after N seconds", retrying if it has retries.
  What the handler returns after the timeout is ignored.

A handler stuck in a synchronous loop cannot see the signal; the
supervisor's backstop still stops it.
@shivaam
shivaam force-pushed the ts-sdk-execution-timeout branch from d1b21fa to 37b77d0 Compare November 3, 2026 03:05
@shivaam

shivaam commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@jason810496 @ChickenBenny rebased on #75387. The runtime now sends SetExecutionTimeout(timeout_seconds) right before the task's work starts, and the merged supervisor accepts it. I reran the full Airflow E2E and got the same results. Could you take another look?


Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting

@ChickenBenny

Copy link
Copy Markdown
Contributor

Hi @shivaam, a correction to my earlier comment. In #76359 I moved the worker-default fallback into the coordinator. When ti_context.execution_timeout is unset, the coordinator fills it in with [core] default_task_execution_timeout and sends it to the runtime in StartupDetails. A 0 stays 0, meaning no timeout. This way no SDK has to read the config or keep its own fallback, so this PR wouldn't need the AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT change I suggested earlier.

The maintainers still need to decide whether this belongs in the supervisor, though, so let's wait for their call before changing anything here.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the update, the direction LGTM now.

Comment on lines +500 to +505
const err = parseFrameError(
await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }),
);
if (err) {
logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code });
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comm.request throws on a closed channel or after 30 s with no reply, so a send failure here fails the task before it starts. Should that warn as well, like the error-frame case?

Suggested change
const err = parseFrameError(
await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }),
);
if (err) {
logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code });
}
try {
const err = parseFrameError(
await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }),
);
if (err) {
logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code });
}
} catch (err) {
logs.warning("Could not send SetExecutionTimeout", {
error: (err as Error).message ?? String(err),
});
}

task.operator,
buildOperatorContext({ details, dag: task.dag, ctx, client, logs, fail }),
),
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A timed-out operator goes through fail and logs "Task failed" without the stack, while a handler logs "Task timed out". Could the catch below match runHandler?

  } catch (err) {
    if (err instanceof TaskTimeoutError) {
      logs.error("Task timed out", { task_id: ctx.taskId, error: err.message, stack: err.stack ?? null });
      return buildFailureResponse(details, err.message);
    }
    return fail((err as Error).message ?? String(err));
  }

if (stopWaiting.aborted) done();
running.then(done, done);
});
throw error;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A handler that ignores the signal keeps running after this throws. The runtime then reports the failure and closes comm, so its later getClient() calls fail on the closed channel, and the process stays up until the supervisor kills it after task_success_overtime. Should the runtime exit once it has reported a timeout?

Comment on lines +83 to +88
if (timeoutMs > MAX_TIMER_MS) return running;

let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = new Promise<typeof TIMED_OUT>((resolve) => {
timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs);
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we return "running" promise if timeoutMs is larger than MAX_TIMER_MS?

Past about 24.8 days the runtime enforces nothing, so a handler that returns after the supervisor's SIGTERM still succeeds. Re-arming the timer in steps keeps the timeout enforced for any length:

Suggested change
if (timeoutMs > MAX_TIMER_MS) return running;
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = new Promise<typeof TIMED_OUT>((resolve) => {
timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs);
});
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = new Promise<typeof TIMED_OUT>((resolve) => {
const arm = (leftMs: number) => {
timer =
leftMs > MAX_TIMER_MS
? setTimeout(() => arm(leftMs - MAX_TIMER_MS), MAX_TIMER_MS)
: setTimeout(() => resolve(TIMED_OUT), leftMs);
};
arm(timeoutMs);
});

Comment on lines +26 to +27
// setTimeout fires at once for a longer delay, so the supervisor alone enforces
// a timeout past this (about 24.8 days).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the timer re-armed, this comment changes too:

Suggested change
// setTimeout fires at once for a longer delay, so the supervisor alone enforces
// a timeout past this (about 24.8 days).
// setTimeout fires at once for a delay past this (about 24.8 days), so a longer
// timeout re-arms the timer in steps of at most this.

@jason810496
jason810496 self-requested a review November 12, 2026 08:28
@ChickenBenny

ChickenBenny commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Hi @shivaam, coming back to the 30s wait after the timeout. I think we can just drop it, without the coordinator having to pass killed_task_cleanup_time in.

SetExecutionTimeout re-arms the supervisor at the same moment the runtime's timer starts, so its SIGTERM lands at exactly timeout + killed_task_cleanup_time. That's the same point you'd get by reading the config. So if wait() just waits for the task or stopWaiting, without cleanupTimer, the runtime gets the right window for free and never has to know the value. Reporting after SIGTERM still works too, because the supervisor keeps handling frames while it escalates to SIGKILL.

This would also line up with the other SDKs:

  • Python: the timeout fires inside the task at T, and the runtime adds no grace of its own. The task gets until SIGTERM at T + killed_task_cleanup_time.
  • Go: I'm planning the same thing for the follow-up to Go SDK: send execution_timeout and use the worker's default #76359. Cancel the task's ctx at T, then wait for the task or SIGTERM.

Right now, with the default of 60s, a TS task gets 30s to clean up while Python and Go get 60s, and changing the config doesn't move the TS number. The docs line "up to 30 seconds" would then become "up to killed_task_cleanup_time".

The 30s self-exit after a plain SIGTERM (#69079) is a separate thing, so I'd leave that as it is.

What do you think?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants