Skip to content

Restore LIMIT 1 on single-row XCom lookups - #72554

Merged
pierrejeambrun merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-xcom-restore-limit-1
Sep 8, 2026
Merged

pierrejeambrun merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-xcom-restore-limit-1

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

Why

How

  • Add .limit(1) back on the four single-row lookups: get_mapped_xcom_by_index and get_xcom, TaskInstance.xcom_pull, and the create_xcom_entry.

Verification

Set up:

uv sync --package apache-airflow-core
uv pip install -e shared/*/

Unit test:

uv run --package apache-airflow-core pytest \
    airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_xcoms.py \
    airflow-core/tests/unit/models/test_taskinstance.py \
    airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_xcom.py

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't found the regression in last PR, nice catch!
The PR LGTM, no further concern from me.

@pierrejeambrun pierrejeambrun left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True that the 2.0 .first() API changed and isn't applying the limit anymore. I wasn't aware of that.

Thanks for the fix.

I think there are two other call site that could use the same update:

  • execution_api/routes/task_instances.py:1228
  • core_api/routes/public/dags.py:443

I'll open a follow up PR for those two thanks.

@pierrejeambrun
pierrejeambrun merged commit 39a5b93 into apache:main Sep 8, 2026
304 checks passed
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 39a5b93 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

@FrankYang0529
FrankYang0529 deleted the airflow-xcom-restore-limit-1 branch September 8, 2026 11:56
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Sep 8, 2026
Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.
pierrejeambrun added a commit that referenced this pull request Sep 8, 2026
Signed-off-by: PoAn Yang <payang@apache.org>

(cherry picked from commit 39a5b93)

Co-authored-by: PoAn Yang <payang@apache.org>
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Sep 8, 2026
Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.
pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Sep 9, 2026
Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.
pierrejeambrun added a commit that referenced this pull request Sep 9, 2026
* Bound remaining single-row lookups with .limit(1)

Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of #52325's migration: get_previous_task_instance
came in #59712, unfavorite_dag's existence probe in #51264. Same
anti-pattern as the four sites #72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.

* Also assert the returned row on the previous-TI regression test
vatsrahul1001 added a commit that referenced this pull request Sep 10, 2026
* Bound remaining single-row lookups with .limit(1)

Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of #52325's migration: get_previous_task_instance
came in #59712, unfavorite_dag's existence probe in #51264. Same
anti-pattern as the four sites #72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.

* Also assert the returned row on the previous-TI regression test

(cherry picked from commit ff83957)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
Signed-off-by: PoAn Yang <payang@apache.org>
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
* Bound remaining single-row lookups with .limit(1)

Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.

* Also assert the returned row on the previous-TI regression test
regarmukesh3g pushed a commit to regarmukesh3g/airflow that referenced this pull request Sep 27, 2026
Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants