Skip to content

Preserve unrelated errors during pending asset run creation - #75523

Open
ataulmujeeb-cyber wants to merge 1 commit into
apache:mainfrom
ataulmujeeb-cyber:apdr-preserve-unrelated-errors
Open

ataulmujeeb-cyber wants to merge 1 commit into
apache:mainfrom
ataulmujeeb-cyber:apdr-preserve-unrelated-errors

Conversation

@ataulmujeeb-cyber

@ataulmujeeb-cyber ataulmujeeb-cyber commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Related: #71074

AssetManager._get_or_create_apdr treats every IntegrityError raised while creating a pending AssetPartitionDagRun as "another producer inserted this row first". Only one constraint actually means that, so unrelated database errors get pulled into the race-recovery path.

Root Cause

The recovery branch catches the base exception class:

except exc.IntegrityError:
    winner = cls._get_latest_pending_apdr(...)

Any constraint failure reaches it. The damaging case is when the error comes from the caller's own pending rows rather than from the new one, because Session.begin_nested() flushes pending ORM state before opening its savepoint:

  1. A caller queues a row that violates NOT NULL, then calls into asset-change registration.
  2. begin_nested() flushes it, the database rejects it, and the transaction is now in a failed state.
  3. Recovery assumes a lost insert race and issues its lookup for the winner.
  4. That lookup runs on the failed transaction, so SQLAlchemy raises PendingRollbackError.

The original error is replaced by one that points somewhere else, and an error that retrying cannot fix is retried first.

Failure Current behaviour Expected
Pending-marker unique violation recover, re-read the winner recover
Unrelated NOT NULL / PK / other UNIQUE recover, error obscured propagate unchanged

Fix

Classify the error from driver diagnostics instead of the rendered message, and recover only for the pending-marker constraint:

except exc.IntegrityError as e:
    if not _is_pending_marker_conflict(e):
        raise

_is_pending_marker_conflict reads SQLSTATE 23505 plus the constraint name on PostgreSQL, sqlite_errorcode 2067 with the column list on SQLite, and errno 1062 with the trailing for key name on MySQL.

Diagnostics rather than text matters because the rendered IntegrityError includes the statement and its bound parameters. An ordinary row whose value happens to contain the constraint name, or a MySQL value carrying its own for key '...' fragment, is enough to make an unrelated failure look like the marker conflict.

Genuine marker conflicts keep the existing two-attempt recovery and the existing exhaustion behaviour of re-raising the driver's IntegrityError. No migration, no schema change.

Changes

  • airflow-core/src/airflow/assets/manager.py: added _is_pending_marker_conflict and narrowed the recovery branch in _get_or_create_apdr to that constraint.
  • airflow-core/tests/unit/assets/test_marker_conflict_classifier.py: new module, 11 tests driven by real database errors:
    • genuine pending-marker conflict recognised on each backend
    • duplicate primary key, NOT NULL and unrelated UNIQUE violations all propagate
    • adversarial values containing the constraint name, and a MySQL value containing a misleading for key '...' fragment
    • helper-level propagation, so an unrelated error escapes _get_or_create_apdr on the first attempt
    • caller-flush failure preserved rather than replaced by PendingRollbackError

Testing

pytest airflow-core/tests/unit/assets/test_marker_conflict_classifier.py
  • PostgreSQL 16 / psycopg3: 11 passed, 0 skipped
  • MySQL 8.4 / PyMySQL: 11 passed, 0 skipped

Earlier runs of the same change on an equivalent tree: 22 passed on psycopg3 for the wider focused selection; 11 passed on psycopg2, mysqlclient and PyMySQL each; 21 passed on SQLite with one pre-existing concurrency test skipped.

Restoring the bare except IntegrityError makes the two negative helper tests fail on PostgreSQL/psycopg3, MySQL/mysqlclient and SQLite: the caller-flush case performs three lookups instead of one and surfaces PendingRollbackError in place of the original error.

The tests provoke real constraint violations rather than constructing exceptions. test_helper_recovers_from_a_real_conflict_end_to_end forces the first lookup to miss so the insert hits a genuine unique violation; it is a staged single-transaction control, not a two-transaction race. Obscuring the original error is a diagnostic problem and these tests do not demonstrate data loss. SQLite results do not establish concurrent-writer behaviour.

Affected Components

  • AssetManager._get_or_create_apdr
  • Partitioned asset consumers using batch_asset_events

Related Follow-ups

Each stands alone and can be reviewed independently.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant