Repository navigation
Conversation
26b04fb to
d1b21fa
Compare
|
Hi @shivaam, thanks for doing this! Some context on where #75387 landed, from @jason810496's reviews (#75387 (review) and #75387 (review)):
This PR also covers what @Andrushika raised in #74869 (#74869 (comment)): language SDK tasks had nothing telling them about the timeout, so they only got SIGTERM at Compared with the above, a few things:
Otherwise looks good to me 👍 Related: #74716, #74717, #74869, #75387 Drafted-by: Claude Code (Opus 5.5); reviewed by @ChickenBenny before posting |
jason810496
left a comment
There was a problem hiding this comment.
Yes, we should follow the direction that @ChickenBenny pointed out, thanks.
|
Thanks @ChickenBenny and @jason810496, agreed. I'll rebase once #75387 merges and send On the 30 s wait: agreed, we can use
Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting |
A TypeScript task got no signal at its execution_timeout: the runtime had no timer, so the task ran on until the supervisor's backstop sent SIGTERM killed_task_cleanup_time later, and a handler that then returned normally was recorded as a success, skipping its retries. The runtime now handles the timeout itself, as the Python task runner does: - It reads execution_timeout from the run context (unset or 0: none; negative: already passed, so the handler does not run). - Right before the handler or operator runs, it sends SetExecutionTimeout with that timeout, so the supervisor's backstop uses the same value and counts from the same moment. - At the timeout it aborts getContext().signal with a new exported TaskTimeoutError, waits up to 30 seconds for the handler to clean up (less if the supervisor's SIGTERM comes first), then fails the task with "Task timed out after N seconds", retrying if it has retries. What the handler returns after the timeout is ignored. A handler stuck in a synchronous loop cannot see the signal; the supervisor's backstop still stops it.
d1b21fa to
37b77d0
Compare
|
@jason810496 @ChickenBenny rebased on #75387. The runtime now sends Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting |
|
Hi @shivaam, a correction to my earlier comment. In #76359 I moved the worker-default fallback into the coordinator. When The maintainers still need to decide whether this belongs in the supervisor, though, so let's wait for their call before changing anything here. |
jason810496
left a comment
There was a problem hiding this comment.
Nice, thanks for the update, the direction LGTM now.
| const err = parseFrameError( | ||
| await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }), | ||
| ); | ||
| if (err) { | ||
| logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code }); | ||
| } |
There was a problem hiding this comment.
comm.request throws on a closed channel or after 30 s with no reply, so a send failure here fails the task before it starts. Should that warn as well, like the error-frame case?
| const err = parseFrameError( | |
| await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }), | |
| ); | |
| if (err) { | |
| logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code }); | |
| } | |
| try { | |
| const err = parseFrameError( | |
| await comm.request({ type: "SetExecutionTimeout", timeout_seconds: timeoutSeconds }), | |
| ); | |
| if (err) { | |
| logs.warning("Supervisor did not take SetExecutionTimeout", { error: err.code }); | |
| } | |
| } catch (err) { | |
| logs.warning("Could not send SetExecutionTimeout", { | |
| error: (err as Error).message ?? String(err), | |
| }); | |
| } |
| task.operator, | ||
| buildOperatorContext({ details, dag: task.dag, ctx, client, logs, fail }), | ||
| ), | ||
| ); |
There was a problem hiding this comment.
A timed-out operator goes through fail and logs "Task failed" without the stack, while a handler logs "Task timed out". Could the catch below match runHandler?
} catch (err) {
if (err instanceof TaskTimeoutError) {
logs.error("Task timed out", { task_id: ctx.taskId, error: err.message, stack: err.stack ?? null });
return buildFailureResponse(details, err.message);
}
return fail((err as Error).message ?? String(err));
}| if (stopWaiting.aborted) done(); | ||
| running.then(done, done); | ||
| }); | ||
| throw error; |
There was a problem hiding this comment.
A handler that ignores the signal keeps running after this throws. The runtime then reports the failure and closes comm, so its later getClient() calls fail on the closed channel, and the process stays up until the supervisor kills it after task_success_overtime. Should the runtime exit once it has reported a timeout?
| if (timeoutMs > MAX_TIMER_MS) return running; | ||
|
|
||
| let timer: ReturnType<typeof setTimeout> | undefined; | ||
| const timedOut = new Promise<typeof TIMED_OUT>((resolve) => { | ||
| timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs); | ||
| }); |
There was a problem hiding this comment.
Why do we return "running" promise if timeoutMs is larger than MAX_TIMER_MS?
Past about 24.8 days the runtime enforces nothing, so a handler that returns after the supervisor's SIGTERM still succeeds. Re-arming the timer in steps keeps the timeout enforced for any length:
| if (timeoutMs > MAX_TIMER_MS) return running; | |
| let timer: ReturnType<typeof setTimeout> | undefined; | |
| const timedOut = new Promise<typeof TIMED_OUT>((resolve) => { | |
| timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs); | |
| }); | |
| let timer: ReturnType<typeof setTimeout> | undefined; | |
| const timedOut = new Promise<typeof TIMED_OUT>((resolve) => { | |
| const arm = (leftMs: number) => { | |
| timer = | |
| leftMs > MAX_TIMER_MS | |
| ? setTimeout(() => arm(leftMs - MAX_TIMER_MS), MAX_TIMER_MS) | |
| : setTimeout(() => resolve(TIMED_OUT), leftMs); | |
| }; | |
| arm(timeoutMs); | |
| }); |
| // setTimeout fires at once for a longer delay, so the supervisor alone enforces | ||
| // a timeout past this (about 24.8 days). |
There was a problem hiding this comment.
With the timer re-armed, this comment changes too:
| // setTimeout fires at once for a longer delay, so the supervisor alone enforces | |
| // a timeout past this (about 24.8 days). | |
| // setTimeout fires at once for a delay past this (about 24.8 days), so a longer | |
| // timeout re-arms the timer in steps of at most this. |
|
Hi @shivaam, coming back to the 30s wait after the timeout. I think we can just drop it, without the coordinator having to pass
This would also line up with the other SDKs:
Right now, with the default of 60s, a TS task gets 30s to clean up while Python and Go get 60s, and changing the config doesn't move the TS number. The docs line "up to 30 seconds" would then become "up to The 30s self-exit after a plain SIGTERM (#69079) is a separate thing, so I'd leave that as it is. What do you think? |
A TypeScript task gets no signal at its
execution_timeout. The runtime has no timer, so the task runs until the supervisor's backstop sends SIGTERMkilled_task_cleanup_time(60 s) later. If the handler then returns normally, the task is recorded as success and its retries are skipped.The TypeScript runtime now handles the timeout itself, as the Python task runner does with SIGALRM:
execution_timeoutfrom the run context. Unset or0means no timeout, as for a Python task. A negative value has already passed, so the handler does not run.SetExecutionTimeoutwith that timeout, so the supervisor's backstop uses the same value and counts from the same moment.getContext().signalwith a new exportedTaskTimeoutErroras itsreason. It then waits up to 30 s (ABORT_GRACE_PERIOD_MS) for the handler to clean up, or less if the supervisor's SIGTERM comes first. Then it fails the task withTask timed out after N seconds, asRetryTaskif it has retries left. What the handler returns after the timeout is ignored, so a timed-out task can no longer succeed.execution_timeout=2,retries=1Those rows are from running a bundle built from this branch, and one built from
main, under the real Python supervisor (_PopenActivitySubprocess, real sockets and signals; only the Execution API was mocked).Not in this PR, as both need the coordinator to pass a value to the runtime: falling back to the worker's
default_task_execution_timeoutwhen the run context has no timeout, and waitingkilled_task_cleanup_timeinstead of a fixed 30 s after the timeout.Validation:
verify:packagepassed; prek hooks passed on commit.main) under the real Python supervisor; after the rebase, the branch cases rerun against the merged supervisor give the same results, and it acceptsSetExecutionTimeout.NodeCoordinator, SQLite): one TypeScript Dag packed withairflow-ts-packagainst this branch and againstmain. Onmain, the task that returns after the abort ends success with its retry skipped; with this change both tries fail with "Task timed out after 3 seconds". A handler that passes the signal stops at the timeout instead of at the supervisor's SIGTERM, and a synchronous loop is still killed by the supervisor.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines