Skip to content

Fix KubernetesJobOperator RBAC and Kueue preemption issues - #74497

Closed
priyanshujaiz wants to merge 3 commits into
apache:mainfrom
priyanshujaiz:fix-kueue-preemption
Closed

priyanshujaiz wants to merge 3 commits into
apache:mainfrom
priyanshujaiz:fix-kueue-preemption

Conversation

@priyanshujaiz

Copy link
Copy Markdown

Closes #74373

Description

Strict RBAC policies in Kubernetes environments may deny access to the /status subresource, causing KubernetesJobOperator to fail with 403 Forbidden when checking job status after upgrading to provider version 9.0.0.

Additionally, jobs preempted by Kueue may be suspended rather than failed. Without handling this state, the operator can poll indefinitely for pods that no longer exist or incorrectly report task completion before the job has finished.

This PR addresses both issues by using the primary /jobs endpoint for status checks and introducing a readmission wait loop. The operator and trigger now handle suspended jobs gracefully, wait for readmission, and rediscover replacement pods so that execution can resume correctly and logs remain accessible.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude Code] following [the guidelines]


  • Read the [Pull Request Guidelines] for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal ([AIP]) is needed.
  • When adding dependency, check compliance with the [ASF 3rd Party License Policy].
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in [airflow-core/newsfragments]. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Oct 9, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this XCom path also needs to participate in the new readmission flow.

Right now the trigger reads XCom from self.pod_names before it waits for the Job with return_on_suspension=True. If a Kueue-managed Job is preempted and those initially discovered pods are deleted, this can fail or hang on the old pod before the trigger gets a chance to detect suspension, wait for readmission, and rediscover the replacement pod.

Could we move XCom extraction after the readmission/job-completion loop, using the latest pod names instead? It would also be good to cover this with a do_xcom_push=True preemption test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Moved XCom extraction after the job-completion loop so it only runs after successful completion, avoiding stale pod references during preemption and readmission. Added test_run_preemption_with_xcom_push_does_not_read_stale_pods to verify that no stale-pod XCom operations occur on the preemption path.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.
I think the same ordering issue remains in the non-deferrable KubernetesJobOperator.execute() path, which extracts XCom from self.pods before the suspension-aware Job polling.
Could we defer that extraction until the replacement pods have completed, and add a synchronous do_xcom_push=True preemption regression test?

Priyanshu Jaiz and others added 2 commits October 17, 2026 16:03
Strict RBAC policies in many Kubernetes environments deny access to the /status subresource, causing operators to fail with 403 Forbidden when checking job status. Additionally, when jobs are preempted by Kueue, they are marked as suspended rather than failed. Without handling this suspended state gracefully, the operator either stalls indefinitely polling missing pods or incorrectly reports premature completion, leading to broken downstream pipelines and inaccessible logs.
When a Kueue-managed Job is preempted and readmitted, the deferrable
path had two bugs:

1. execute_complete fetched the job status using self.name/self.namespace,
   which hold the operator template name rather than the concrete Kubernetes
   resource name (job-<base>-<random-suffix>). This would target the wrong
   Job when the name was auto-generated. The fix reads the identity directly
   from the trigger event, which always carries the fully-resolved name and
   namespace.

2. KubernetesJobTrigger.run() extracted XCom from self.pod_names before
   entering the job-completion loop. If Kueue preempted the Job and deleted
   those pods, the sidecar wait would hang or 404 before the trigger could
   detect suspension. The fix moves XCom extraction to after the loop, so
   it only runs against the live pods that successfully executed the Job,
   and never runs at all on the preemption/readmission path.

Adds regression tests for both fixes, including the missing
do_xcom_push=True preemption test.
self.job = self.hook.get_job_status(
job_name=self.job.metadata.name, namespace=self.job.metadata.namespace
)
self.pods = self.get_pods(pod_request_obj=self.pod_request_obj, context=context)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can race with pod creation after readmission too.
The loop exits as soon as spec.suspend becomes false, but the replacement pod may not be discoverable yet, so get_pods() can raise No pods running.
Maybe we can wait or retry pod discovery here.

if self.wait_until_job_complete:
self.pods: Sequence[k8s.V1Pod] = self.get_pods(
pod_request_obj=self.pod_request_obj, context=context
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this initial get_pods() can fail before the suspension-aware polling begins.
A Kueue-managed Job can be suspended before a pod is created, or preempted before discovery, so this can raise No pods running instead of waiting for readmission.
Maybe we can check for suspension before pod discovery and wait for a pod to become available?

self.job = self.hook.get_job_status(job_name=event["name"], namespace=event["namespace"])
self.job_request_obj = self.build_job_request_obj(context)
self.pod_request_obj = self.build_pod_request_obj(context)
self.pods = self.get_pods(pod_request_obj=self.pod_request_obj, context=context)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@priyanshujaiz

Copy link
Copy Markdown
Author

Thanks for the review and feedback. After investigating the issue further, I wasn't able to arrive at a correct and reliable solution. Rather than leave this PR open with an incomplete implementation, I'll close it for now.

I appreciate the maintainers' time and feedback. I'll revisit the problem if I'm able to develop a more robust solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesStartKueueJobOperator: completion wait ignores Kueue preemption and requires the jobs/status subresource

2 participants