Skip to content

Handle missed deadline alerts when DagRuns fail - #68961

Closed
shivaam wants to merge 13 commits into
apache:mainfrom
shivaam:codex/airflow-60927-deadline-failure
Closed

shivaam wants to merge 13 commits into
apache:mainfrom
shivaam:codex/airflow-60927-deadline-failure

Conversation

@shivaam

@shivaam shivaam commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor
  • Add DeadlineAlert.fire_on_failure, defaulting to False.
  • When the scheduler marks a DagRun failed, immediately handle only pending deadlines whose alert opted into fire_on_failure=True.
  • Keep the default behavior unchanged: deadlines without fire_on_failure=True still fire only when deadline_time is reached.
  • Keep manual UI/API mark-failed behavior out of scope.
  • Keep deadline-alert errors from blocking DagRun finalization, while allowing DB errors to use the scheduler retry path.

Closes: #60927

Details

Failed DagRuns currently leave pending deadline alerts untouched until the scheduler's expired-deadline loop reaches deadline_time. For long deadline intervals, that can delay the alert by hours after the DagRun has already failed.

This PR adds an opt-in flag, DeadlineAlert.fire_on_failure, for users who want deadline alerts to fire as soon as an automated scheduler-driven DagRun failure is finalized.

The default is False for backward compatibility. Existing Deadline Alerts keep their current behavior and fire at deadline_time.

The independent callback data persistence bug was split into #69073 and has already merged, so this PR is now focused on the failure-time opt-in behavior only.

Also tested with a local Airflow cluster:
fire_on_failure=True sync/executor callback fired immediately on DagRun failure.
fire_on_failure=True async/triggerer callback fired immediately on DagRun failure.
fire_on_failure=False stayed pending after failure and fired only after deadline_time.

Tests

Local focused validation passed:

  • DagRun deadline failure behavior: 6 passed
  • Task SDK deadline tests: 21 passed
  • DeadlineAlert serialization/model/private-UI coverage: 11 passed
  • Focused Ruff check/format: passed

Also tested with a real local Airflow/Postgres daemon harness:

  • fire_on_failure=True sync/executor callback fired immediately on DagRun failure.
  • fire_on_failure=True async/triggerer callback fired immediately on DagRun failure.
  • fire_on_failure=False stayed pending after failure and fired only after deadline_time.

Was generative AI tooling used to co-author this PR?
  • Yes — Codex

Important

🛠️ Maintainer triage note for @shivaam · by @potiuk · 2026-07-08 15:38 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Pre-commit / static checks failing (CI image checks / Static checks). See the contributor guide.

Full criteria: Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@shivaam
shivaam requested review from XD-DENG and ashb as code owners June 25, 2026 03:17
@boring-cyborg boring-cyborg Bot added the area:deadline-alerts AIP-86 (former AIP-57) label Jun 25, 2026
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from 5299bc4 to bf27fda Compare June 25, 2026 03:38
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from bf27fda to 48d9021 Compare June 25, 2026 03:54
@potiuk

potiuk commented Jun 25, 2026

Copy link
Copy Markdown
Member

@shivaam — the Static checks CI job is failing here, which needs a code fix on your side (a rerun won't clear it). You can reproduce and fix locally with:

pre-commit run --all-files     # or: breeze static-checks

Once those pass and CI is green, it'll be ready for a maintainer to pick up. Thanks!

See the PR quality criteria.

Automated first-pass triage note drafted by an AI-assisted tool — may get things wrong; once addressed, a real Apache Airflow maintainer takes the next look. (why automated)


Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from 48d9021 to 3eb066b Compare June 25, 2026 17:09
@eladkal
eladkal requested a review from ferruzzi June 26, 2026 03:48
@ferruzzi

Copy link
Copy Markdown
Contributor

Thanks for the PR and the thorough testing. This had a pretty long discussion in the Issue and I still don't think short-circuiting by default is the right answer.

Let's say you have a Dag which generates a report which you need for your morning meeting so you set a Deadline Alert for 9AM which emails you saying your report won't be ready.

Case 1) You work at Big Company. The Dag fails at 7AM, an admin sees this, determines it's just a network hiccup and clears the failing tasks which then pass at 7:30. If the deadline fired on failure, you would have your report and a notice saying you won't.

Case 2) Your Dag may have an on_failure_callback configured which notifies you of the failure. If the deadline also fires immediately, you now get two notifications about the same event with different messaging: Schrodinger's Dag is both failed and late.

Case 3) You work at Small Company and don't have an admin team, you do it all yourself, so a Dag failing likely won't be cleared and the missed deadline is inevitable. I'd argue that on_failure_callback currently handles this, but I can see the advantage of bundling it into the Deadline Alert directly..

Case 4) You realize something is wrong with the input and you manually mark the Dag as FAILED so the report doesn't get generated with bad data. Are you going to fix the data and re-trigger the Dag, or is that the end of the story? We don't know and shouldn't guess. Worth noting: this PR wouldn't cover this case either, which would introduce a difference in behavior between an automatic failure and a manual one.

Overall, always short-circuiting is not the answer. I would propose three alternatives which are not mutually exclusive:

  1. Add a flag to the Deadline Alert class. Naming is hard, but something like fire_on_failure: bool = False and let the user decide depending on their use case.
  2. Add a hook in the Deadline class which accepts a run_id and prunes all future deadlines for that run. This would allow on_failure_callback (or any other callback) to take action and then call the new hook to remove the now-redundant deadline(s) so you don't get double-notified.
  3. Anywhere in the UI where a user can mark a Dag as FAILED should have an option to also clear related Deadlines.

To be clear, I do agree that there is a real user need here, I'm not trying to kill this or dissuade you. I just want to make sure we solve it without breaking the existing use cases. It's one very valid use case, but that doesn't invalidate the others.

That said, here's my suggestion: I'd also ask that the callback.data persistence fixes in deadline.py be split into a separate PR since those fix a real bug independent of this feature question. Whatever we do about the failure state, that looks like an actual legit improvement/bug-fix. Then I'd suggest reworking this PR to implement the flag and tracking the other two options as new Issues.

((tagging @ramitkataria since he was also involved in the original discussion and may have thoughts.))

@shivaam

shivaam commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

@ferruzzi thanks, this makes sense. I split the independent callback.data persistence fix out into #69073 so it can be reviewed separately from the failure-behavior question. That PR is scoped to Deadline.handle_miss() persisting the callback context/routing data correctly.

For this PR, I agree that always firing the Deadline Alert on failure is too broad as the default. I will rework this toward an opt-in DeadlineAlert flag (name TBD, maybe something like fire_on_failure / alert_on_failure, defaulting to False) so users can choose this behavior for the cases where a failed run means the deadline is inevitably missed.

The prune-hook and UI/manual-failure options also make sense to me, but I think those can be tracked as follow-up issues unless you think they should be included in this PR.

@potiuk
potiuk marked this pull request as draft July 2, 2026 17:38
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from 3eb066b to 4bf50ea Compare July 2, 2026 18:33
@ferruzzi

ferruzzi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

The bugfix portion is approved and merged; let me know when you are ready for a re-review on this one or if you want to discuss it more, otherwise I'll just let you work on it in peace. 👍

@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from 4bf50ea to 9e0d761 Compare July 2, 2026 22:53
@shivaam
shivaam marked this pull request as ready for review July 2, 2026 23:30
@shivaam

shivaam commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

@amoghrajesh / @kaxil If you get a chance, could you please help review this PR

@eladkal eladkal added this to the Airflow 3.4.0 milestone Jul 26, 2026
@eladkal eladkal added the type:improvement Changelog: Improvements label Jul 26, 2026
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch 3 times, most recently from 05edb4e to 3a6fd94 Compare August 19, 2026 13:53
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from 3a6fd94 to e19f682 Compare August 27, 2026 14:13
@eladkal

eladkal commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

@shivaam can you rebase and resolve conflicts?

@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from e19f682 to f09963c Compare September 11, 2026 05:22
@shivaam

shivaam commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@eladkal Rebased onto the latest main and resolved the conflicts. CI is now green, and the PR is mergeable. Thanks!

Keep the checked-in UI client byte-for-byte aligned with the current generator after rebasing onto main.
Scheduler-enforced timeouts bypass normal Dag-run state evaluation, so they need the same opt-in failure handling as task and deadlock failures.
Keep the new opt-in behavior distinct from the three parameters every deadline alert requires.
Protect SQLite foreign keys while the deadline alert table is rebuilt and keep the failure-path regression tests deterministic and concise.
Older SDK deadline alerts do not expose the new failure flag, so serialization must retain the false default. SQLAlchemy failures while handling missed deadlines must propagate so the surrounding transaction can roll back and retry safely.
Regenerate the unmerged fire_on_failure migration through Airflow's supported workflow so its revision metadata has verifiable provenance. Preserve the existing schema operations and SQLite foreign-key safety while synchronizing the migration references.
The deadline alert response now includes the opt-in failure flag, so representative UI fixtures must include its backward-compatible false default.
Autospecced method mocks ensure signature changes cannot silently bypass the failure-path coverage.
The upstream migration chain advanced while this change was awaiting review, so the deadline alert migration must follow the new head to preserve a single upgrade path.
@shivaam
shivaam force-pushed the codex/airflow-60927-deadline-failure branch from b22e36f to 72e4f0d Compare September 29, 2026 07:40
@eladkal

eladkal commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@shivaam tests are failing

Dag version comparisons must recognize failure-alert policy changes without reporting differences for legacy definitions that use the default policy.
description: Interval in seconds between the reference time and the deadline.
Null for a dynamic interval (e.g. a VariableInterval) whose value is only
resolved at scheduler evaluation time.
fire_on_failure:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand why this setting is needed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fire_on_failure lets a deadline alert fire immediately when its DagRun fails, even if the deadline hasn’t expired. It defaults to False to preserve existing behavior.

This particular YAML change exposes that policy in the API response for a future UI addition—for example, showing “Also alert on DagRun failure” alongside the deadline details, so users understand why an alert fired before its configured deadline.

The current UI doesn’t consume this field yet, so we can omit it from the private UI response for now and add it alongside that UI change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is that this feels more like a bug fix rather than a feature hence no need for a flag/setting.
cc @ferruzzi

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is a new feature but yeah it is in grey area between feature and bug fix. @ferruzzi let me know what think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I was out sick most of the week and just catching up. Here's my opinion and my reasoning. My intent was fail-early, I would like to drop the flag and call this a bug-fix. But that is not how the feature is documented. The doc pages say (some quotes, not exhaustive):

  • "This example will send an email notification if the Dag hasn't finished 30 minutes after it was queued." (Core Concepts)
  • "would trigger the alert if the Dag hasn't completed 15 minutes after it was scheduled to start, regardless of when (or if) it actually began executing." (how-to, DAGRUN_LOGICAL_DATE)
  • "Alert if 30 minutes past average runtime", "Dag is running longer than expected!" (how-to, AVERAGE_RUNTIME)

Which means a user going off of only what they can see in the docs would expect that failure is a "finished" state and the callback shouldn't fire. That's what the Issue claims, and it's a fair reading of the docs.

To test this, I set Claude a task: "There is some interesting discussion around the Deadline Alerts feature in Airflow. I want you to answer this question using ONLY official public airflow doc pages, do not use my design notes, code comments, open github issues, web searches, etc. I want a blind reading direct from official documentation as a new user might have, not as an insider with design and implementation knowledge. If I create a Dag with a DeadlineAlert whose interval is three hours and that Dag fails after ten minutes, what is the expected behavior? Would the callback fire after 10 minutes, 3 hours, or never? Form your answer in the format Setting a Deadline Alert could be read as "If this Dag _____ before this time, then fire this callback".

and it replied with: "If this Dag has not finished before this time, then fire this callback"` It would never fire. That is the only one of the three a doc-only reader can reasonably land on.

Which implies that regardless of my intention, the documentation as written doesn't convey what I meant for it to say.

If we rephrase the docs to say "if this Dag has not succeeded before this time, then do this callback", this is closer to what I had in mind and what Shivaam implemented. I'd say that was a bug-fix toward my vision, but if we're being fair, it's moving away from the documented behavior and the docs should be rephrased accordingly from "finished" to "succeeded", etc. But, regardless of my intention, I think I have to admit that a fair reading of the docs would be "do not fire on failure" which makes it hard to argue that this is a bug-fix and why I backed the flag.

Now, If we're open to changing the default behavior, then I'd actually propose inverting the flag. Make it work as-intended (fire on fail by default) with an opt-out (treat failure as "finished"; prune on failure and do not fire the callback). Users could then combine Deadlines with on_failure_callback or other methods of monitoring Dag failures to distinguish between "ran long" and "failed", and they don't necessarily want duplicate alarms on failure. Which I think is the use-case that caused me to implement it this way in the first place, but now it's just causing confusion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am adding a third approach below :)

My understanding is that failure notifications and deadline notifications serve different purposes. on_failure_callback tells users that a Dag has failed; a deadline alert tells them that the expected outcome wasn’t achieved by the deadline. I’d prefer keeping that distinction explicit so users know they would need both.

Under that approach, deadline alerts would retain their deadline-based timing. Users who already receive failure notifications could optionally suppress the pending deadline alert after failure, perhaps through a flag such as suppress_deadline_on_failure. Users wanting immediate failure notifications would use on_failure_callback.

For example, a Dag could fail at 7 a.m. with a 9 a.m. deadline. The on-call receives the failure notification, clears the failed tasks, and attempts recovery. If the pending deadline alert was cancelled at 7 a.m., they would lose the notification that recovery still hadn’t succeeded by 9 a.m.

That leaves a question: should clearing the failed tasks restore deadline monitoring? From an on-call perspective, I would want that coverage restored while attempting recovery.
Would this separation make sense, rather than making deadline callbacks fire immediately on failure?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won’t have the bandwidth to rework the implementation further, but we can settle the behavior in this PR so whoever picks it up has a clear direction.

@shivaam shivaam closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:deadline-alerts AIP-86 (former AIP-57) ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. type:improvement Changelog: Improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadline Alert triggered after dag failure

5 participants