Repository navigation
CMP-4178: Add machineset scale up after remediations applied - #72
Anna-Koudelkova wants to merge 5 commits into
Conversation
|
/retest |
|
|
||
| err = helpers.ValidateMachineSetScaleWithDeletePolicyNewest(tc, c, platformBindingName) | ||
| if err != nil { | ||
| t.Fatalf("MachineSet scale/deletePolicy validation failed: %s", err) |
There was a problem hiding this comment.
Should we use t.Errorf() for these error checks?
There was a problem hiding this comment.
That makes sense, thank you! I have originally used tFatalf() because the whole test cases uses it, but we probably do not need the test to fail immediately on failure in these added steps.
|
/retest |
1 similar comment
|
/retest |
|
Hi Anna, Looked over the PR and everything seems fine to me. There's one small bug that Claude picked up on which, while it doesn't break the testing, is a quick fix for correctness' sake (that avoids an unlikely, though possible edge case):
In practice this is unlikely since the validation runs after MCP rollout completes and all nodes should be Ready, but it's still reading the wrong field. Suggest changing line 1877 from reading originalReplicas, foundReplicas, _ := unstructured.NestedInt64(msObj.Object, "spec", "replicas")
if !foundReplicas {
originalReplicas = 1
}...Defaulting to 1 rather than 0, since the That being said, I think this is a minor issue and shouldn't be blocking for the PR, so /lgtm |
|
/lgtm |
ValidateMachineSetScaleWithDeletePolicyNewest took status.readyReplicas as the MachineSet's original size. That field is a transient observation, and the helper samples it immediately after ApplyRemediationsWithDependencies, i.e. while MCO may still be rebooting workers, so readyReplicas is often below spec.replicas. When that happens the test is both useless and destructive: with spec.replicas=3 and readyReplicas=2, the "scale up" writes replicas=3, which is a no-op that satisfies the wait as soon as the rebooting node comes back, and the "scale down" then writes replicas=2 and permanently removes a real worker. A single-replica MachineSet whose node is briefly NotReady reads as readyReplicas=0 and gets scaled to zero. Take the baseline from spec.replicas, which is the only field that states the desired size, and treat a missing spec.replicas as an error rather than defaulting to zero. Gate the whole flow on the MachineSet being converged (status.replicas and status.readyReplicas both matching spec.replicas) so an unsettled MachineSet fails fast with a clear message instead of silently burning the scale-up timeout. Checking status.replicas as well as readyReplicas also catches a machine that is already being provisioned or deleted. The snapshot is now a struct; it had outgrown a five-value return. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cleanup defer is registered before anything scales the MachineSet, so a failure in setMachineSetDeletePolicyNewest or verifyMachineSetDeletePolicyNewest -- a 409 conflict is realistic here, since the object was fetched several Update round-trips earlier -- still caused the deferred cleanup to write spec.replicas on a MachineSet the test had never touched. Track whether the scale-up was attempted and skip the replicas restore when it was not. The flag is set before scaleMachineSetAndWait rather than after it, because that call can fail with spec.replicas already written, and that case does need restoring. The deletePolicy restore and the label removal still run unconditionally: the labels are always applied by that point, and rewriting deletePolicy to the value it already has is harmless. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every write in the machineset scale test read its object well before it wrote it, and gave up on the first conflict. The worst case was setMachineSetDeletePolicyNewest: the object it updated was fetched before labelExistingMachinesForMachineSet ran N sequential Update round-trips against the same MachineSet's machines, so the resourceVersion was very likely stale by the time it was written. The machine-API controllers write to these objects whenever a machine is created, becomes ready, or goes away, which is continuous during the post-remediation reboots that immediately precede this test. A conflict there failed the whole suite for a reason that has nothing to do with compliance. Add updateWithConflictRetry, which reads, mutates and writes in one place and re-reads on conflict, and route the deletePolicy set, both scale writes, the cleanup replicas and deletePolicy restores, and both machine label updates through it. client-go's retry.RetryOnConflict is not vendored and client-go v0.30.0 is not in the module cache, so this uses the same shape (5 attempts, 10ms apart) without a new dependency. Dropping the passed-in object also removes the unused msKey parameter from setMachineSetDeletePolicyNewest and the now-unused obj field on machineSetState, and lets cleanup distinguish a machine that the scale-down legitimately removed from a real failure to unlabel it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verifyOriginalMachinesPreserved could not fail. It listed machines immediately after scaleMachineSetAndWait returned, and that returns as soon as status.readyReplicas is back to the original count -- which the MachineSet controller satisfies the moment it stamps a deletionTimestamp on its victim. The Machine object itself survives the whole drain and deprovision, minutes later. So in the exact regression this test exists to catch, the controller deleting an older machine instead of the newest, the victim was still returned by the List and the check passed. It also passed on a healthy cluster, for the same reason: the assertion returned nil either way. Treat a machine with a deletionTimestamp as not preserved, and turn the check from "nothing went missing" into a positive assertion. Capture the name of the machine the scale-up added while it is still the only new one, then require that specific machine to be the one going away and every original to still be live. The check now polls, because the controller may not have picked a victim at all when readyReplicas first drops, and fails fast via backoff.Permanent when an original machine is the one being deleted. Machine listing moves into listMachinesForMachineSet, which splits live from deleting and is shared with the labeling path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@Anna-Koudelkova: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Based on a discussion we had with Xiaojie and Taimur, it is important to keep a check that machinesets are scalable after applying remediations and the
deletePolicy: Newestwould always delete the newly added machinesets first when asked to scale down.This was previously a customer bug.