[ATTA-E-MUSTAFA]

Patch on top of 4.575.v6969b_7c435eb_, the version our controller runs (branch starts at that tag; the diff below is only the patch).

Problem

Our PR builds can run on three clouds that share a label: Toronto, Delhi and Milan, cheapest first. The plugin doesn't use that order:

• NoDelayProvisionerStrategy shuffles the clouds and stops after the first one that serves the label, even when that cloud planned nothing.
• When GCP rejects the VM create (ZONE_RESOURCE_POOL_EXHAUSTED, QUOTA_EXCEEDED, ...), ComputeEngineComputerLauncher only logs it. The dead agent stays, Jenkins counts it as capacity on its way, and the build waits on that cloud until it times out.

Change

• Config order. Clouds are tried in their configured order (no shuffle), so the cheapest one comes first.
• Remember rejections. New CloudFailureTracker records a cloud when GCP rejects one of its VMs: an insert operation error other than OPERATION_CANCELED_BY_USER (aborted builds), or an IOException from the insert call itself. Such a cloud is tried after all other clouds for 5 minutes, then goes back to its place. Configure with -Dcom.google.jenkins.plugins.computeengine.CloudFailureTracker.cooldownMinutes=N. In memory only, so a restart resets it.
• Drop the dead agent. After a rejected create the agent is removed from Jenkins (GCP created no VM), so the waiting build is provisioned again on the next NodeProvisioner run, about 10s later, from the next cloud.
• Fall through. A cloud that plans no node (at its instance cap, or refused) no longer ends the search; the next cloud is tried in the same run.

Flow on a stockout

1. A build waits for pr-pool. Order is Toronto, Delhi, Milan, so Toronto is asked.
2. GCP returns ZONE_RESOURCE_POOL_EXHAUSTED. Toronto is recorded and the agent is removed.
3. On the next provisioning run (~10s) the order is Delhi, Milan, Toronto. Delhi creates the VM.
4. 5 minutes later Toronto is first again.

Effect on today's config

Every label in our jenkins.yaml is served by exactly one cloud, so ordering and cooldown change nothing until clouds share a label. The one immediate difference: an agent whose create GCP rejected is removed right away instead of lingering.

Testing

• Unit test CloudFailureTrackerTest: healthy by default, cooldown per cloud, recovery after the cooldown.
• Not built yet. Run mvn -Dchangelist=575.educative-1 package (e.g. in maven:3.9-eclipse-temurin-17) and fix any spotless/spotbugs findings.
• On the controller, install 4.575.educative-1 and confirm builds behave as today.
• Add a shared pr-pool label to pr, pr-1, pr-2, plus Testing-Haider first with a machine type GCP rejects. Check the system log shows Cloud gce-Testing-Haider could not create a VM (...) and the build lands on a pr-* agent within ~20s.

Rollout notes

• Install via config/plugins.txt in educative/jenkins: google-compute-engine:4.575.educative-1:<release .hpi URL>.
• Don't update this plugin from the Jenkins UI; that replaces the patch.
• When Jenkins is upgraded (4.703+ needs Jenkins 2.516.3), port this onto the newer plugin, which also has native fallbackZones.

🤖 Generated with Claude Code