Skip to content

Bound remaining single-row lookups with .limit(1) - #72699

Merged
pierrejeambrun merged 2 commits into
apache:mainfrom
astronomer:fix-remaining-unbounded-first-lookups
Sep 9, 2026
Merged

pierrejeambrun merged 2 commits into
apache:mainfrom
astronomer:fix-remaining-unbounded-first-lookups

Conversation

@pierrejeambrun

@pierrejeambrun pierrejeambrun commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #72554, and depends on it. That PR restores .limit(1) on the
four single-row XCom lookups whose LIMIT was dropped when the code migrated
from the legacy Query.first() to SQLAlchemy 2.0's Result.first() (which
does not add a LIMIT of its own — see the Result.first docstring). A wider
audit of the remaining .first() calls under airflow/api_fastapi/ turned
up two more sites with the same shape:

  • execution_api/routes/task_instances.py::get_previous_task_instance — reads
    one previous TI ordered by logical_date DESC, no LIMIT. Without the bound
    the query pulls every earlier matching TI into the Python process and returns
    the first. Introduced in Add ability to get previous TI on RuntimeTaskInstance #59712.
  • core_api/routes/public/dags.py::unfavorite_dag — probes dag_favorite for
    the current (dag_id, user_id) before deletion. The schema bounds the result
    to a single row, so the practical cost here is small, but the pattern is the
    same. Introduced in Add support for favorite/pin dags to dashboard #51264.

The regression tests reuse the capture_orm_selects helper #72554 adds to
devel-common, so this branch is stacked on top of that PR and should be
merged after it.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@FrankYang0529 FrankYang0529 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.

@pierrejeambrun pierrejeambrun changed the title Bound remaining single-row XCom-style lookups with .limit(1) Bound remaining single-row lookups with .limit(1) Sep 8, 2026
Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.
@pierrejeambrun
pierrejeambrun force-pushed the fix-remaining-unbounded-first-lookups branch from 4464b4f to 9c4a9c2 Compare September 9, 2026 12:17
@pierrejeambrun
pierrejeambrun force-pushed the fix-remaining-unbounded-first-lookups branch from 9c4a9c2 to 3b62671 Compare September 9, 2026 13:55
@pierrejeambrun pierrejeambrun added this to the Airflow 3.3.2 milestone Sep 9, 2026
@pierrejeambrun
pierrejeambrun merged commit ff83957 into apache:main Sep 9, 2026
79 checks passed
@pierrejeambrun
pierrejeambrun deleted the fix-remaining-unbounded-first-lookups branch September 9, 2026 16:21
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker ff83957 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

vatsrahul1001 added a commit that referenced this pull request Sep 10, 2026
* Bound remaining single-row lookups with .limit(1)

Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of #52325's migration: get_previous_task_instance
came in #59712, unfavorite_dag's existence probe in #51264. Same
anti-pattern as the four sites #72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.

* Also assert the returned row on the previous-TI regression test

(cherry picked from commit ff83957)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
* Bound remaining single-row lookups with .limit(1)

Two more callers of ``session.scalars(...).first()`` /
``session.execute(select(...)).first()`` were still missing an explicit
``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add
one (unlike the legacy ``Query.first``), each call sends every matching
row to the Python process and discards all but the first. The favorite
existence check is bounded to a single row per (dag_id, user_id) by the
schema, but the previous-TI lookup scans every earlier matching TI.

Introduced independently of apache#52325's migration: get_previous_task_instance
came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same
anti-pattern as the four sites apache#72554 fixes; the tests reuse the
``capture_orm_selects`` helper that PR introduces.

* Also assert the returned row on the previous-TI regression test
@ashb

ashb commented Sep 14, 2026

Copy link
Copy Markdown
Member

@pierrejeambrun @jason810496 aren't a good chunk of these better as query.one_or_none()?

@jason810496

Copy link
Copy Markdown
Member

@pierrejeambrun @jason810496 aren't a good chunk of these better as query.one_or_none()?

We will need to check the behavior of SQLA2 on the one_or_none method I guess.
Not sure does it apply the limit under the hook.

@FrankYang0529

Copy link
Copy Markdown
Member

@pierrejeambrun @jason810496 aren't a good chunk of these better as query.one_or_none()?

We will need to check the behavior of SQLA2 on the one_or_none method I guess. Not sure does it apply the limit under the hook.

IIUC, both session.execute(query) or session.scalars(query) run the statement as written before one_or_none() is even called (session.py#L2373-L2380, session.py#L2481-L2488). The first form calls Result.one_or_none(), and the second calls ScalarResult.one_or_none(). Both pass the same arguments to _only_one_row() (result.py#L1474-L1497, result.py#L1847-L1857). The _only_one_row() function doesn't change the SQL.

@jason810496

jason810496 commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Thanks @FrankYang0529 for the follow-up so we will keep them as-is (keep the limit(1)) instead of changing them all to one_or_none() the right? Thanks.

@FrankYang0529

Copy link
Copy Markdown
Member

Thanks @FrankYang0529 for the follow-up so we will keep them as-is (keep the limit(1)) instead of changing them all to one_or_none() the right? Thanks.

Yes, I'd keep the .limit(1). The legacy Query.one_or_none() doesn't add a LIMIT either. It just calls self._iter().one_or_none() (query.py#L2792), and only Query.first() adds self.limit(1) (query.py#L2762-L2766).

@pierrejeambrun

pierrejeambrun commented Sep 16, 2026 •

Copy link
Copy Markdown
Member Author

Yep, db scan cost + transfert time. On the db side there is no limit applied for one_or_none too. The cursor will discard the buffer and stop after the 2nd row matching, but that can still be waste of resources. (even with one_or_none, we would need a limit(2) apparently to make sure the request is bounded.

@shahar1 shahar1 linked an issue Nov 6, 2026 that may be closed by this pull request
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ti.get_previous_ti method is not optimized

5 participants