Skip to content

Keep operator extra links viewable for previous task tries - #73315

Open
amoghrajesh wants to merge 3 commits into
apache:mainfrom
astronomer:handle-links-for-tries
Open

amoghrajesh wants to merge 3 commits into
apache:mainfrom
astronomer:handle-links-for-tries

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #71471


Was generative AI tooling used to co-author this PR?
  • Yes claude sonnet

What

Operator extra links are stored in xcom under one key per task instance, and every retry deletes the task's xcoms before running. An extra link records what one attempt did, so the next attempt destroying it means earlier tries lose their link.

The extra links endpoint and the UI already pass try_number end to end, but XComOperatorLink.get_link ignored it and read the single surviving row, so the API looked try-aware while silently serving the latest try's link for every try.

Current behaviour

A task that fails once and succeeds on retry leaves a single xcom row holding the second attempt's link. Opening try 1 in the UI shows try 2's link, with nothing indicating it belongs to a different attempt. If the retry dies before finalize() runs, no link exists at all.

This predates Airflow 3. Airflow 2 links read XCom for the runtime IDs they needed (job id, run id, cluster name) and lost them to the same issue. The rewrite to XComOperatorLink carried the gap forward rather than introducing it, but also linked a single finished string written at a known point, which is what makes a per-try key cheap now.

Proposed change

Each attempt writes its link under a key of its own, _link_<LinkClass>__try_<n>, and the retry wipe leaves operator link keys alone. The resolver looks up the key for the try being viewed and falls back to the unsuffixed key when there is no per try row.

Changes of Note

Filtering happens on the worker, not the API server. The server lists which xcom keys exist; the worker already holds the task object and knows its link keys, so no Dag parsing is added to the ti_run path and the execution API contract is untouched.

The legacy row is kept, not wiped. After upgrading, a row written before this change survives and acts as the fallback for tries that have no per try row. Old tries keep showing what they show today rather than losing their button.

User implications / backcompat

No schema change, no migration, no config.

  • Runs that executed before this change resolve through the fallback and render exactly as they do now.
  • A new API server with an older worker degrades to current behaviour: the worker writes only the unsuffixed key, the server falls back to it.
  • A task with several tries now has one xcom row per try per link instead of one. Link rows are not currently hidden from the xcom tab, so they are more visible there.

Testing

Unit tests cover the resolver reading the requested try, the fallback to the unsuffixed key, the per try write in finalize(), and the wipe skipping link keys while still clearing everything else. Each fails without the source change.

Manual validation

Using this DAG:

import pendulum

from airflow.sdk import DAG, BaseOperator
from airflow.sdk.bases.operatorlink import BaseOperatorLink


class JobConsoleLink(BaseOperatorLink):
    name = "Job console"

    def get_link(self, operator, *, ti_key):
        return f"https://example.com/job?try={ti_key.try_number}"


class FlakyJobOperator(BaseOperator):
    operator_extra_links = (JobConsoleLink(),)

    def execute(self, context):
        if context["ti"].try_number == 1:
            raise RuntimeError("boom on first try")
        return "ok"


with DAG(
    "test_link_per_try",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule=None,
    catchup=False,
):
    FlakyJobOperator(task_id="run_job", retries=1, retry_delay=pendulum.duration(seconds=5))
Simple situation: Clean database

Try 1:

image

Try 2:

image image
Situation where old format was in xcom already: newly upgraded
image

Try 1 and 2 were already run so they both show older behaviour: showing link with try=2

image image

On upgrade to this new version, the task is cleared and third try is launched and it has link with try=3:

image image

Upgrade and mixed version cases checked by hand:

  • Link written before this change, then both server and worker upgraded. The unsuffixed row is still resolved through the fallback and renders as it did before, for every try.
  • Such a run cleared and re-run after upgrading. The new attempt gets its own per try link; earlier tries fall back to the pre upgrade row and keep their button rather than going blank.
  • New API server with an older worker. The worker writes only the unsuffixed key, the server falls back to it, and behaviour matches current Airflow.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@amoghrajesh

Copy link
Copy Markdown
Contributor Author

@ashb would you mind taking a look at this?

@staticmethod
def _read_value(session: Session, key: str, ti_key: TaskInstanceKey) -> Row | None:
return session.execute(
XComModel.get_many(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't there a .get_one()?

@amoghrajesh amoghrajesh Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no get_one on XComModel, although its present in the docstring 🤣. Removing it while I am at it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +1575 to +1577
if is_link_xcom_key(x, link_xcom_keys):
# skip clearing this key as it is an operator link
continue

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be done on the server side, not the client side I feel (i.e. server shouldn't tell client to clear these)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks. Moved it to the server. ti_run now resolves the task's link xcom_keys from the cached serdag and leaves them out of xcom_keys_to_clear, so the client just deletes what it is given.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

)
assert mock_xcom_set.mock_calls == [
call(
key="_link_AirflowLink__try_1",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the storage tab in the UI already filter out the se ^_link_AirflowLink items?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the storage tab doesn't filter these.

Which did you have in mind -
hide all _link_* rows, or only the new _link_*__try_N ones?

The narrower rule keeps the tab identical to today for existing runs...

@amoghrajesh
amoghrajesh requested a review from ashb September 29, 2026 09:56
.limit(1)
).first()
# Runs from before per-try keys existed only have the unsuffixed key.
result = self._read_value(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #74222 landed on main, I think most of this is already handled there. Each retry now runs under a fresh TI UUID and ti_run only clears XComs that the current attempt produced, so an earlier try's link rows aren't touched. On the read side, XComOperatorLink.get_link already filters on try_number=ti_key.try_number.

Is anything left here once you rebase? This branch conflicts with main now. The bit I couldn't work out is the unsuffixed fallback for rows written before the upgrade; that would depend on how #74222 maps xcom_v1 rows to a try number.

@vatsrahul1001 vatsrahul1001 added the backport-to-v3-4-test Backport to v3-4-test label Oct 18, 2026
@pierrejeambrun

pierrejeambrun commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Ping for this, do we still need this? Woud be great to have that links working again for 3.4.0

@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Oct 20, 2026
@bbovenzi bbovenzi added the area:UI Related to UI/UX. For Frontend Developers. label Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dag-processor area:task-sdk area:UI Related to UI/UX. For Frontend Developers. backport-to-v3-4-test Backport to v3-4-test type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Operator extra links don't work per attempt

7 participants