Skip to content

Fix airflow db clean foreign key violation on dag_version - #68523

Closed
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-db-clean-fk-violation-dag-version
Closed

potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-db-clean-fk-violation-dag-version

Conversation

@potiuk

@potiuk potiuk commented Jun 14, 2026

Copy link
Copy Markdown
Member

airflow db clean aborted with a ForeignKeyViolation when an old dag_version row scoped for deletion was still referenced by a newer task instance (or dag run) that was not itself in scope:

psycopg2.errors.ForeignKeyViolation: update or delete on table "dag_version" violates foreign key constraint "task_instance_dag_version_id_fkey" on table "task_instance"
DETAIL:  Key (id)=(0198edf9-f9f1-7d73-92c1-fac5359c10b7) is still referenced from table "task_instance".

db clean cleans tables in dependency order but filters each table by its own recency column, so an old dag_version (old created_at, not the latest for its dag) gets scoped for deletion while a newer task instance still referencing it is not. task_instance.dag_version_id is ON DELETE RESTRICT (so the delete raises) and dag_run.created_dag_version_id is ON DELETE SET NULL (so the delete would silently null a surviving run's reference).

This adds a skip_if_referenced_by option to the cleanup table config. dag_version rows whose id still appears in task_instance.dag_version_id or dag_run.created_dag_version_id are excluded from deletion; they are removed on a later run once the referencing rows have themselves aged out of scope. Tests cover both the skip path and that an unreferenced old version is still deleted.

Thanks to @HsiuChuanHsu for the reproduction and the original approach in #56567.

closes: #56192


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

`airflow db clean` aborted with a ForeignKeyViolation when an old
`dag_version` row scoped for deletion was still referenced by a newer
task instance (or dag run) that was not itself in scope for deletion.

`dag_version` cleanup now skips rows whose id is still referenced from
`task_instance.dag_version_id` (ON DELETE RESTRICT) or
`dag_run.created_dag_version_id` (ON DELETE SET NULL). Such rows are
deleted on a later run once the referencing rows have themselves aged
out of scope.
@potiuk potiuk added this to the Airflow 3.3.0 milestone Jun 14, 2026
@ephraimbuddy

Copy link
Copy Markdown
Contributor

Looks like my PR: #68339 is fixing the same issue

@potiuk

potiuk commented Jun 14, 2026

Copy link
Copy Markdown
Member Author

Ha Ha.. Approved it then . Easy to miss - seems another issues described the same ;)

@potiuk

potiuk commented Jun 14, 2026

Copy link
Copy Markdown
Member Author

Superseded by #68339

@potiuk potiuk closed this Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ForeignKeyViolation error when doing airflow db clean

2 participants