Repository navigation
Conversation
`airflow db clean` aborted with a ForeignKeyViolation when an old `dag_version` row scoped for deletion was still referenced by a newer task instance (or dag run) that was not itself in scope for deletion. `dag_version` cleanup now skips rows whose id is still referenced from `task_instance.dag_version_id` (ON DELETE RESTRICT) or `dag_run.created_dag_version_id` (ON DELETE SET NULL). Such rows are deleted on a later run once the referencing rows have themselves aged out of scope.
Contributor
|
Looks like my PR: #68339 is fixing the same issue |
Member
Author
|
Ha Ha.. Approved it then . Easy to miss - seems another issues described the same ;) |
Member
Author
|
Superseded by #68339 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
airflow db cleanaborted with aForeignKeyViolationwhen an olddag_versionrow scoped for deletion was still referenced by a newer task instance (or dag run) that was not itself in scope:db cleancleans tables in dependency order but filters each table by its own recency column, so an olddag_version(oldcreated_at, not the latest for its dag) gets scoped for deletion while a newer task instance still referencing it is not.task_instance.dag_version_idisON DELETE RESTRICT(so the delete raises) anddag_run.created_dag_version_idisON DELETE SET NULL(so the delete would silently null a surviving run's reference).This adds a
skip_if_referenced_byoption to the cleanup table config.dag_versionrows whoseidstill appears intask_instance.dag_version_idordag_run.created_dag_version_idare excluded from deletion; they are removed on a later run once the referencing rows have themselves aged out of scope. Tests cover both the skip path and that an unreferenced old version is still deleted.Thanks to @HsiuChuanHsu for the reproduction and the original approach in #56567.
closes: #56192
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines