Repository navigation
Conversation
5299bc4 to
bf27fda
Compare
bf27fda to
48d9021
Compare
|
@shivaam — the Static checks CI job is failing here, which needs a code fix on your side (a rerun won't clear it). You can reproduce and fix locally with: Once those pass and CI is green, it'll be ready for a maintainer to pick up. Thanks! See the PR quality criteria. Automated first-pass triage note drafted by an AI-assisted tool — may get things wrong; once addressed, a real Apache Airflow maintainer takes the next look. (why automated) Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting |
48d9021 to
3eb066b
Compare
|
Thanks for the PR and the thorough testing. This had a pretty long discussion in the Issue and I still don't think short-circuiting by default is the right answer. Let's say you have a Dag which generates a report which you need for your morning meeting so you set a Deadline Alert for 9AM which emails you saying your report won't be ready. Case 1) You work at Big Company. The Dag fails at 7AM, an admin sees this, determines it's just a network hiccup and clears the failing tasks which then pass at 7:30. If the deadline fired on failure, you would have your report and a notice saying you won't. Case 2) Your Dag may have an Case 3) You work at Small Company and don't have an admin team, you do it all yourself, so a Dag failing likely won't be cleared and the missed deadline is inevitable. I'd argue that Case 4) You realize something is wrong with the input and you manually mark the Dag as FAILED so the report doesn't get generated with bad data. Are you going to fix the data and re-trigger the Dag, or is that the end of the story? We don't know and shouldn't guess. Worth noting: this PR wouldn't cover this case either, which would introduce a difference in behavior between an automatic failure and a manual one. Overall, always short-circuiting is not the answer. I would propose three alternatives which are not mutually exclusive:
To be clear, I do agree that there is a real user need here, I'm not trying to kill this or dissuade you. I just want to make sure we solve it without breaking the existing use cases. It's one very valid use case, but that doesn't invalidate the others. That said, here's my suggestion: I'd also ask that the ((tagging @ramitkataria since he was also involved in the original discussion and may have thoughts.)) |
|
@ferruzzi thanks, this makes sense. I split the independent For this PR, I agree that always firing the Deadline Alert on failure is too broad as the default. I will rework this toward an opt-in The prune-hook and UI/manual-failure options also make sense to me, but I think those can be tracked as follow-up issues unless you think they should be included in this PR. |
3eb066b to
4bf50ea
Compare
|
The bugfix portion is approved and merged; let me know when you are ready for a re-review on this one or if you want to discuss it more, otherwise I'll just let you work on it in peace. 👍 |
4bf50ea to
9e0d761
Compare
|
@amoghrajesh / @kaxil If you get a chance, could you please help review this PR |
05edb4e to
3a6fd94
Compare
3a6fd94 to
e19f682
Compare
|
@shivaam can you rebase and resolve conflicts? |
e19f682 to
f09963c
Compare
|
@eladkal Rebased onto the latest main and resolved the conflicts. CI is now green, and the PR is mergeable. Thanks! |
Keep the checked-in UI client byte-for-byte aligned with the current generator after rebasing onto main.
Scheduler-enforced timeouts bypass normal Dag-run state evaluation, so they need the same opt-in failure handling as task and deadlock failures.
Keep the new opt-in behavior distinct from the three parameters every deadline alert requires.
Protect SQLite foreign keys while the deadline alert table is rebuilt and keep the failure-path regression tests deterministic and concise.
Older SDK deadline alerts do not expose the new failure flag, so serialization must retain the false default. SQLAlchemy failures while handling missed deadlines must propagate so the surrounding transaction can roll back and retry safely.
Regenerate the unmerged fire_on_failure migration through Airflow's supported workflow so its revision metadata has verifiable provenance. Preserve the existing schema operations and SQLite foreign-key safety while synchronizing the migration references.
The deadline alert response now includes the opt-in failure flag, so representative UI fixtures must include its backward-compatible false default.
Autospecced method mocks ensure signature changes cannot silently bypass the failure-path coverage.
The upstream migration chain advanced while this change was awaiting review, so the deadline alert migration must follow the new head to preserve a single upgrade path.
b22e36f to
72e4f0d
Compare
|
@shivaam tests are failing |
Dag version comparisons must recognize failure-alert policy changes without reporting differences for legacy definitions that use the default policy.
| description: Interval in seconds between the reference time and the deadline. | ||
| Null for a dynamic interval (e.g. a VariableInterval) whose value is only | ||
| resolved at scheduler evaluation time. | ||
| fire_on_failure: |
There was a problem hiding this comment.
I am not sure I understand why this setting is needed?
There was a problem hiding this comment.
fire_on_failure lets a deadline alert fire immediately when its DagRun fails, even if the deadline hasn’t expired. It defaults to False to preserve existing behavior.
This particular YAML change exposes that policy in the API response for a future UI addition—for example, showing “Also alert on DagRun failure” alongside the deadline details, so users understand why an alert fired before its configured deadline.
The current UI doesn’t consume this field yet, so we can omit it from the private UI response for now and add it alongside that UI change.
There was a problem hiding this comment.
My point is that this feels more like a bug fix rather than a feature hence no need for a flag/setting.
cc @ferruzzi
There was a problem hiding this comment.
I think it is a new feature but yeah it is in grey area between feature and bug fix. @ferruzzi let me know what think.
There was a problem hiding this comment.
Sorry, I was out sick most of the week and just catching up. Here's my opinion and my reasoning. My intent was fail-early, I would like to drop the flag and call this a bug-fix. But that is not how the feature is documented. The doc pages say (some quotes, not exhaustive):
- "This example will send an email notification if the Dag hasn't finished 30 minutes after it was queued." (Core Concepts)
- "would trigger the alert if the Dag hasn't completed 15 minutes after it was scheduled to start, regardless of when (or if) it actually began executing." (how-to, DAGRUN_LOGICAL_DATE)
- "Alert if 30 minutes past average runtime", "Dag is running longer than expected!" (how-to, AVERAGE_RUNTIME)
Which means a user going off of only what they can see in the docs would expect that failure is a "finished" state and the callback shouldn't fire. That's what the Issue claims, and it's a fair reading of the docs.
To test this, I set Claude a task: "There is some interesting discussion around the Deadline Alerts feature in Airflow. I want you to answer this question using ONLY official public airflow doc pages, do not use my design notes, code comments, open github issues, web searches, etc. I want a blind reading direct from official documentation as a new user might have, not as an insider with design and implementation knowledge. If I create a Dag with a DeadlineAlert whose interval is three hours and that Dag fails after ten minutes, what is the expected behavior? Would the callback fire after 10 minutes, 3 hours, or never? Form your answer in the format Setting a Deadline Alert could be read as "If this Dag _____ before this time, then fire this callback".
and it replied with: "If this Dag has not finished before this time, then fire this callback"` It would never fire. That is the only one of the three a doc-only reader can reasonably land on.
Which implies that regardless of my intention, the documentation as written doesn't convey what I meant for it to say.
If we rephrase the docs to say "if this Dag has not succeeded before this time, then do this callback", this is closer to what I had in mind and what Shivaam implemented. I'd say that was a bug-fix toward my vision, but if we're being fair, it's moving away from the documented behavior and the docs should be rephrased accordingly from "finished" to "succeeded", etc. But, regardless of my intention, I think I have to admit that a fair reading of the docs would be "do not fire on failure" which makes it hard to argue that this is a bug-fix and why I backed the flag.
Now, If we're open to changing the default behavior, then I'd actually propose inverting the flag. Make it work as-intended (fire on fail by default) with an opt-out (treat failure as "finished"; prune on failure and do not fire the callback). Users could then combine Deadlines with on_failure_callback or other methods of monitoring Dag failures to distinguish between "ran long" and "failed", and they don't necessarily want duplicate alarms on failure. Which I think is the use-case that caused me to implement it this way in the first place, but now it's just causing confusion.
There was a problem hiding this comment.
I am adding a third approach below :)
My understanding is that failure notifications and deadline notifications serve different purposes. on_failure_callback tells users that a Dag has failed; a deadline alert tells them that the expected outcome wasn’t achieved by the deadline. I’d prefer keeping that distinction explicit so users know they would need both.
Under that approach, deadline alerts would retain their deadline-based timing. Users who already receive failure notifications could optionally suppress the pending deadline alert after failure, perhaps through a flag such as suppress_deadline_on_failure. Users wanting immediate failure notifications would use on_failure_callback.
For example, a Dag could fail at 7 a.m. with a 9 a.m. deadline. The on-call receives the failure notification, clears the failed tasks, and attempts recovery. If the pending deadline alert was cancelled at 7 a.m., they would lose the notification that recovery still hadn’t succeeded by 9 a.m.
That leaves a question: should clearing the failed tasks restore deadline monitoring? From an on-call perspective, I would want that coverage restored while attempting recovery.
Would this separation make sense, rather than making deadline callbacks fire immediately on failure?
There was a problem hiding this comment.
I won’t have the bandwidth to rework the implementation further, but we can settle the behavior in this PR so whoever picks it up has a clear direction.
DeadlineAlert.fire_on_failure, defaulting toFalse.fire_on_failure=True.fire_on_failure=Truestill fire only whendeadline_timeis reached.Closes: #60927
Details
Failed DagRuns currently leave pending deadline alerts untouched until the scheduler's expired-deadline loop reaches
deadline_time. For long deadline intervals, that can delay the alert by hours after the DagRun has already failed.This PR adds an opt-in flag,
DeadlineAlert.fire_on_failure, for users who want deadline alerts to fire as soon as an automated scheduler-driven DagRun failure is finalized.The default is
Falsefor backward compatibility. Existing Deadline Alerts keep their current behavior and fire atdeadline_time.The independent callback data persistence bug was split into #69073 and has already merged, so this PR is now focused on the failure-time opt-in behavior only.
Also tested with a local Airflow cluster:
fire_on_failure=True sync/executor callback fired immediately on DagRun failure.
fire_on_failure=True async/triggerer callback fired immediately on DagRun failure.
fire_on_failure=False stayed pending after failure and fired only after deadline_time.
Tests
Local focused validation passed:
6 passed21 passed11 passedAlso tested with a real local Airflow/Postgres daemon harness:
fire_on_failure=Truesync/executor callback fired immediately on DagRun failure.fire_on_failure=Trueasync/triggerer callback fired immediately on DagRun failure.fire_on_failure=Falsestayed pending after failure and fired only afterdeadline_time.Was generative AI tooling used to co-author this PR?
Important
🛠️ Maintainer triage note for @shivaam · by
@potiuk· 2026-07-08 15:38 UTCHelpful heads-up from the maintainers — please address before this PR can be reviewed:
CI image checks / Static checks). See the contributor guide.Full criteria: Pull Request quality criteria.
The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.
Automated triage — may be imperfect; a maintainer takes the next look.