Skip to content

Support asynchronous task-clear operations with progress and status tracking #77201

Open

Description

@safaehar

Description

Support durable, asynchronous task-clear operations: the API accepts a clear request quickly, returns an operation ID, and Airflow processes it in the background while the UI shows its status/progress. The intended user experience is similar to submitting and monitoring a backfill.

Today, users can wait for a synchronous clear request only to receive an HTTP 500 when a database statement times out. Acceptance of a clear request should be separate from completion of the clear itself.

Use case/motivation

On Airflow 3.3.2 with PostgreSQL, we observed task-clear requests returning HTTP 500 at our configured 10-second statement_timeout, with the dag_run update waiting on another transaction's lock.

The scheduler provides a contention path: it locks a batch of RUNNING DagRuns, checks them sequentially, and commits after the batch. Clearing tasks also updates their DagRun rows, so a clear can wait behind scheduling work for other runs in that batch. Our batch limit was 120, above the default of 20. The blocking backend in the captured database wait was not conclusively identified as a scheduler; this is motivation for a more resilient clear workflow, rather than proof that every timeout has that cause.

Relevant 3.3.2 code:

Source inspection of main at 3b35135afe9549e99088c4d557c62e5deafb125b also showed the batch-level commit. We have not reproduced the API timeout on that revision.

Proposed behavior

  1. The user confirms a task clear. The API checks authorization, validates the request, and durably records the operation without waiting to mutate the target DagRun/task rows.
  2. Return promptly, for example 202 Accepted with an operation ID and status URL. The UI shows that the clear is queued, rather than implying it has already completed.
  3. A background processor performs the clear while preserving existing task/DagRun locking and consistency guarantees.
  4. The UI can poll or subscribe to status such as queued, in progress, completed, or failed, with progress and actionable errors for large operations.

Moving the same database transaction into an async def endpoint would not provide this behavior. The operation must survive API process restarts and outlive the original HTTP request. Background execution must also handle transient lock/statement timeouts with rollback and bounded retries/backoff, and expose terminal failure if it cannot complete; asynchronous acceptance alone does not remove database contention.

Please consider safe retry/idempotency semantics so client retries or worker restarts do not apply a clear twice. The operation should define its target selection and how it handles tasks whose state or attempt changes after submission, including partial progress across multiple runs.

An opt-in asynchronous mode could preserve compatibility for existing clients that expect synchronous completion. Scheduler lock-duration improvements would be complementary; the primary request here is asynchronous clear submission and tracking.

Suggested acceptance tests

  • Hold a target DagRun lock in another transaction: submitting the asynchronous clear still returns an operation ID promptly, before that lock is released.
  • Release the lock: background processing eventually completes the clear and updates its status.
  • Restart an API/worker process or retry submission: the request remains discoverable and completed work is not cleared again.
  • Exhaust background retries: the UI reports a terminal failure with a useful reason rather than leaving the operation indefinitely pending.

These are proposed tests, not an executed reproduction.

Related issues

Are you willing to submit a PR?

  • Yes I am willing to submit a PR.

Activity

  1. changed the title [-]Reduce scheduler DagRun lock lifetime to avoid task-clear API timeouts[/-] [+]Support asynchronous task-clear operations with progress and status tracking[/+] on Nov 17, 2026
  2. devnaldo commented on Nov 17, 2026

    @devnaldo

    Hi! I’d like to work on this. I’m reading how the clear endpoints take their DagRun locks and what happens when a lock or statement timeout hits. I’ll start by looking at making that failure a retryable error instead of a 500, and I’ll share a short design note here before touching anything bigger, like the async operation. Happy to coordinate if someone else is already on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions