Skip to content

Fix airflow db clean hang on MySQL when delete fails - #66296

Merged
ephraimbuddy merged 9 commits into
apache:mainfrom
Vamsi-klu:fix-db-clean-mysql-hang-66177
Jul 14, 2026
Merged

ephraimbuddy merged 9 commits into
apache:mainfrom
Vamsi-klu:fix-db-clean-mysql-hang-66177

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented May 3, 2026 •

Copy link
Copy Markdown

Fixes a MySQL hang in airflow db clean --skip-archive when a delete fails on a foreign-key constraint.

On the failure path, _do_delete() now rolls back the session before dropping the temporary archive table, and the drop uses the session's active connection instead of checking out a new pooled connection. This prevents metadata-lock contention from leaving the cleanup command stuck indefinitely.

Also adds:

  • unit coverage for rollback-before-drop ordering
  • a MySQL backend regression test for the deadlock case
  • a Postgres backend regression test for the failure path

Validation:

  • prek run --from-ref main --stage pre-commit
  • prek run --from-ref main --stage manual
  • targeted MySQL and Postgres regression tests in Breeze
  • full airflow-core/tests/unit/utils/test_db_cleanup.py in Breeze (57 passed, 1 skipped)
  • breeze ci selective-check --commit-ref HEAD indicates the shared test utility change expands the CI matrix beyond what was practical to run locally

closes: #66177


Was generative AI tooling used to co-author this PR?
  • Yes — Codex; Claude Code (Fable 5)

Generated-by: Codex; Claude Code (Fable 5) following the guidelines


Important

🛠️ Maintainer triage note for @Vamsi-klu · by @potiuk · 2026-07-02 17:46 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed (see the Pull Request quality criteria):

  • This PR has merge conflicts with main. Please rebase onto the latest main and resolve them.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@csrinivas2130

csrinivas2130 commented May 5, 2026 •

Copy link
Copy Markdown

@Vamsi-klu can you close your PR, I have made the fix 4 days ago and waiting for the #PR-66249 approval for merge and close this issue.

@Vamsi-klu Vamsi-klu closed this May 10, 2026
@Vamsi-klu
Vamsi-klu deleted the fix-db-clean-mysql-hang-66177 branch May 10, 2026 01:00
@Vamsi-klu Vamsi-klu reopened this May 13, 2026
@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from 9967327 to 6e2e0bf Compare May 13, 2026 04:47
@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from 6e2e0bf to 7dec317 Compare May 21, 2026 15:47
@potiuk

potiuk commented May 24, 2026

Copy link
Copy Markdown
Member

@Vamsi-klu A few things need addressing before review — see our Pull Request quality criteria.

  • ❌ CI fails: Low dep tests: providers / All-prov:LowestDeps:14:3.10:airbyte...apprise (and possibly other checks — see the Checks tab for the full list).

No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch 2 times, most recently from 89e743d to 35eb8d4 Compare May 25, 2026 20:24
@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label May 26, 2026
@Vamsi-klu

This comment was marked as spam.

@ephraimbuddy ephraimbuddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will solve the hang but the cleanup would still raise on every dag with run history since dagversion is fk-pinned to task_instance.

The newsfragments is not necessary.

@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from 35eb8d4 to de47db7 Compare June 11, 2026 03:29
@Vamsi-klu

This comment was marked as spam.

@Vamsi-klu

This comment was marked as spam.

@Vamsi-klu

This comment was marked as spam.

@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from de47db7 to f0eecec Compare June 20, 2026 16:33
@potiuk
potiuk marked this pull request as draft July 2, 2026 17:47
@Vamsi-klu
Vamsi-klu marked this pull request as ready for review July 3, 2026 01:45
@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from f0eecec to 7972b0c Compare July 3, 2026 01:45

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the fix! While testing this, I found a small edge case where the cleanup path can still mask the original delete error. Would you mind fixing it in this PR as well? :)

Vamsi-klu added 7 commits July 6, 2026 03:52
The check-newsfragment-pr-number CI job requires the newsfragment
filename to match the PR number, not the linked issue number.
Keeps the hang-regression tests meaningful once dag_version cleanup learns
to skip FK-pinned rows (apache#68339).
When _do_delete unwinds from a failed DELETE, dropping the archive table in
the finally block could itself raise (e.g. a dead connection on MySQL). Python
makes a finally-raised error the top-level exception, so that cleanup error
would replace the original delete failure -- and an OperationalError from the
drop could then be downgraded to a warning by _suppress_with_logging, silently
masking the real IntegrityError. Guard the cleanup so it only propagates on the
success path; on the failure path it is logged and the original error survives.
@Vamsi-klu
Vamsi-klu force-pushed the fix-db-clean-mysql-hang-66177 branch from 7972b0c to 5a98dd5 Compare July 6, 2026 03:52
@Vamsi-klu

This comment was marked as spam.

@Vamsi-klu
Vamsi-klu requested a review from aaron-y-chen July 6, 2026 03:57

@ephraimbuddy ephraimbuddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't have PR numbers on tests doc strings

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
@Vamsi-klu

This comment was marked as spam.

The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal.
@ephraimbuddy
ephraimbuddy merged commit d6cd459 into apache:main Jul 14, 2026
150 checks passed
joshuabvarghese pushed a commit to joshuabvarghese/airflow that referenced this pull request Jul 16, 2026
* Fix airflow db clean hang on MySQL when delete fails (apache#66177)

* Rename newsfragment to match PR number

The check-newsfragment-pr-number CI job requires the newsfragment
filename to match the PR number, not the linked issue number.

* Remove newsfragment per review

* Suppress rollback errors so the original delete error propagates

* Explain why the archive table drop reuses the session connection

* Force the FK-failure path in backend regression tests via config copy

Keeps the hang-regression tests meaningful once dag_version cleanup learns
to skip FK-pinned rows (apache#68339).

* Guard archive-table cleanup so it can't mask the original delete error

When _do_delete unwinds from a failed DELETE, dropping the archive table in
the finally block could itself raise (e.g. a dead connection on MySQL). Python
makes a finally-raised error the top-level exception, so that cleanup error
would replace the original delete failure -- and an OperationalError from the
drop could then be downgraded to a warning by _suppress_with_logging, silently
masking the real IntegrityError. Guard the cleanup so it only propagates on the
success path; on the failure path it is logged and the original error survives.

* Update airflow-core/tests/unit/utils/test_db_cleanup.py

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>

* Retry CI after provider compat startup failure

The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal.

---------

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Co-authored-by: Ramachandra Nalam <nalamvamsi13@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow db clean --tables dag_version --skip-archive hangs forever (MDL deadlock) when an old dag_version is FK-pinned by task_instance

5 participants