Repository navigation
Restore LIMIT 1 on single-row XCom lookups - #72554
Conversation
edd01f7 to
96ebe28
Compare
96ebe28 to
e3a4a76
Compare
Signed-off-by: PoAn Yang <payang@apache.org>
e3a4a76 to
a49b4be
Compare
jason810496
left a comment
There was a problem hiding this comment.
I didn't found the regression in last PR, nice catch!
The PR LGTM, no further concern from me.
There was a problem hiding this comment.
True that the 2.0 .first() API changed and isn't applying the limit anymore. I wasn't aware of that.
Thanks for the fix.
I think there are two other call site that could use the same update:
- execution_api/routes/task_instances.py:1228
- core_api/routes/public/dags.py:443
I'll open a follow up PR for those two thanks.
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 39a5b93 v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of apache#52325's migration: get_previous_task_instance came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same anti-pattern as the four sites apache#72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces.
Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of apache#52325's migration: get_previous_task_instance came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same anti-pattern as the four sites apache#72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces.
Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of apache#52325's migration: get_previous_task_instance came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same anti-pattern as the four sites apache#72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces.
* Bound remaining single-row lookups with .limit(1) Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of #52325's migration: get_previous_task_instance came in #59712, unfavorite_dag's existence probe in #51264. Same anti-pattern as the four sites #72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces. * Also assert the returned row on the previous-TI regression test
* Bound remaining single-row lookups with .limit(1) Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of #52325's migration: get_previous_task_instance came in #59712, unfavorite_dag's existence probe in #51264. Same anti-pattern as the four sites #72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces. * Also assert the returned row on the previous-TI regression test (cherry picked from commit ff83957) Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Signed-off-by: PoAn Yang <payang@apache.org>
* Bound remaining single-row lookups with .limit(1) Two more callers of ``session.scalars(...).first()`` / ``session.execute(select(...)).first()`` were still missing an explicit ``.limit(1)``. Because SQLAlchemy 2.0's ``Result.first()`` does not add one (unlike the legacy ``Query.first``), each call sends every matching row to the Python process and discards all but the first. The favorite existence check is bounded to a single row per (dag_id, user_id) by the schema, but the previous-TI lookup scans every earlier matching TI. Introduced independently of apache#52325's migration: get_previous_task_instance came in apache#59712, unfavorite_dag's existence probe in apache#51264. Same anti-pattern as the four sites apache#72554 fixes; the tests reuse the ``capture_orm_selects`` helper that PR introduces. * Also assert the returned row on the previous-TI regression test
Signed-off-by: PoAn Yang <payang@apache.org>
Why
Query.first()tosession.scalars(...).first()/session.execute(...).first(), and the explicit.limit(1)was dropped in review as redundant withfirst()(Remove Legacy Query Object Usage from airflow-models #52325 (comment)).Query.first(), which addsLIMIT 1to the SQL.Result.first()does not: its docstring says "no limit is applied to the SQL query", and with a buffering DBAPI driver "all rows will be sent to the Python process and all but the first row will be discarded".How
.limit(1)back on the four single-row lookups:get_mapped_xcom_by_indexandget_xcom,TaskInstance.xcom_pull, and thecreate_xcom_entry.Verification
Set up:
Unit test:
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.