Repository navigation
Fix KubernetesJobOperator RBAC and Kueue preemption issues - #74497
priyanshujaiz wants to merge 3 commits into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
There was a problem hiding this comment.
I think this XCom path also needs to participate in the new readmission flow.
Right now the trigger reads XCom from self.pod_names before it waits for the Job with return_on_suspension=True. If a Kueue-managed Job is preempted and those initially discovered pods are deleted, this can fail or hang on the old pod before the trigger gets a chance to detect suspension, wait for readmission, and rediscover the replacement pod.
Could we move XCom extraction after the readmission/job-completion loop, using the latest pod names instead? It would also be good to cover this with a do_xcom_push=True preemption test.
There was a problem hiding this comment.
Fixed. Moved XCom extraction after the job-completion loop so it only runs after successful completion, avoiding stale pod references during preemption and readmission. Added test_run_preemption_with_xcom_push_does_not_read_stale_pods to verify that no stale-pod XCom operations occur on the preemption path.
There was a problem hiding this comment.
Thanks for the update.
I think the same ordering issue remains in the non-deferrable KubernetesJobOperator.execute() path, which extracts XCom from self.pods before the suspension-aware Job polling.
Could we defer that extraction until the replacement pods have completed, and add a synchronous do_xcom_push=True preemption regression test?
Strict RBAC policies in many Kubernetes environments deny access to the /status subresource, causing operators to fail with 403 Forbidden when checking job status. Additionally, when jobs are preempted by Kueue, they are marked as suspended rather than failed. Without handling this suspended state gracefully, the operator either stalls indefinitely polling missing pods or incorrectly reports premature completion, leading to broken downstream pipelines and inaccessible logs.
When a Kueue-managed Job is preempted and readmitted, the deferrable path had two bugs: 1. execute_complete fetched the job status using self.name/self.namespace, which hold the operator template name rather than the concrete Kubernetes resource name (job-<base>-<random-suffix>). This would target the wrong Job when the name was auto-generated. The fix reads the identity directly from the trigger event, which always carries the fully-resolved name and namespace. 2. KubernetesJobTrigger.run() extracted XCom from self.pod_names before entering the job-completion loop. If Kueue preempted the Job and deleted those pods, the sidecar wait would hang or 404 before the trigger could detect suspension. The fix moves XCom extraction to after the loop, so it only runs against the live pods that successfully executed the Job, and never runs at all on the preemption/readmission path. Adds regression tests for both fixes, including the missing do_xcom_push=True preemption test.
90ea417 to
31ea075
Compare
| self.job = self.hook.get_job_status( | ||
| job_name=self.job.metadata.name, namespace=self.job.metadata.namespace | ||
| ) | ||
| self.pods = self.get_pods(pod_request_obj=self.pod_request_obj, context=context) |
There was a problem hiding this comment.
I think this can race with pod creation after readmission too.
The loop exits as soon as spec.suspend becomes false, but the replacement pod may not be discoverable yet, so get_pods() can raise No pods running.
Maybe we can wait or retry pod discovery here.
| if self.wait_until_job_complete: | ||
| self.pods: Sequence[k8s.V1Pod] = self.get_pods( | ||
| pod_request_obj=self.pod_request_obj, context=context | ||
| ) |
There was a problem hiding this comment.
I think this initial get_pods() can fail before the suspension-aware polling begins.
A Kueue-managed Job can be suspended before a pod is created, or preempted before discovery, so this can raise No pods running instead of waiting for readmission.
Maybe we can check for suspension before pod discovery and wait for a pod to become available?
| self.job = self.hook.get_job_status(job_name=event["name"], namespace=event["namespace"]) | ||
| self.job_request_obj = self.build_job_request_obj(context) | ||
| self.pod_request_obj = self.build_pod_request_obj(context) | ||
| self.pods = self.get_pods(pod_request_obj=self.pod_request_obj, context=context) |
|
Thanks for the review and feedback. After investigating the issue further, I wasn't able to arrive at a correct and reliable solution. Rather than leave this PR open with an incomplete implementation, I'll close it for now. I appreciate the maintainers' time and feedback. I'll revisit the problem if I'm able to develop a more robust solution. |
Closes #74373
Description
Strict RBAC policies in Kubernetes environments may deny access to the
/statussubresource, causingKubernetesJobOperatorto fail with403 Forbiddenwhen checking job status after upgrading to provider version 9.0.0.Additionally, jobs preempted by Kueue may be suspended rather than failed. Without handling this state, the operator can poll indefinitely for pods that no longer exist or incorrectly report task completion before the job has finished.
This PR addresses both issues by using the primary
/jobsendpoint for status checks and introducing a readmission wait loop. The operator and trigger now handle suspended jobs gracefully, wait for readmission, and rediscover replacement pods so that execution can resume correctly and logs remain accessible.Was generative AI tooling used to co-author this PR?
Generated-by: [Claude Code] following [the guidelines]
{pr_number}.significant.rst, in [airflow-core/newsfragments]. You can add this file in a follow-up commit after the PR is created so you know the PR number.