Skip to content

CMP-4178: Add machineset scale up after remediations applied - #72

Open
Anna-Koudelkova wants to merge 5 commits into
ComplianceAsCode:mainfrom
Anna-Koudelkova:CMP-4178
Open

Anna-Koudelkova wants to merge 5 commits into
ComplianceAsCode:mainfrom
Anna-Koudelkova:CMP-4178

Conversation

@Anna-Koudelkova

@Anna-Koudelkova Anna-Koudelkova commented Mar 26, 2026 •

Copy link
Copy Markdown
Collaborator

Based on a discussion we had with Xiaojie and Taimur, it is important to keep a check that machinesets are scalable after applying remediations and the deletePolicy: Newest would always delete the newly added machinesets first when asked to scale down.
This was previously a customer bug.

@vickeybrown

Copy link
Copy Markdown
Collaborator

/retest

Comment thread e2e_test.go Outdated

err = helpers.ValidateMachineSetScaleWithDeletePolicyNewest(tc, c, platformBindingName)
if err != nil {
t.Fatalf("MachineSet scale/deletePolicy validation failed: %s", err)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use t.Errorf() for these error checks?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense, thank you! I have originally used tFatalf() because the whole test cases uses it, but we probably do not need the test to fail immediately on failure in these added steps.

@taimurhafeez

Copy link
Copy Markdown
Collaborator

/retest

1 similar comment
@taimurhafeez

Copy link
Copy Markdown
Collaborator

/retest

@abushkin-redhat

abushkin-redhat commented Jul 14, 2026 •

Copy link
Copy Markdown
Collaborator

Hi Anna,

Looked over the PR and everything seems fine to me. There's one small bug that Claude picked up on which, while it doesn't break the testing, is a quick fix for correctness' sake (that avoids an unlikely, though possible edge case):


helpers/utilities.go:1877 — getMachineSetState reads status.readyReplicas to snapshot the original count, but this value is later used to set spec.replicas for scaling and cleanup restoration. These two fields can diverge — spec.replicas is the desired count while status.readyReplicas is the observed count. If a machine happens to
be provisioning or unhealthy when the test runs, the cleanup would permanently shrink the MachineSet to the lower observed count.

In practice this is unlikely since the validation runs after MCP rollout completes and all nodes should be Ready, but it's still reading the wrong field. Suggest changing line 1877 from reading status.readyReplicas to spec.replicas:

originalReplicas, foundReplicas, _ := unstructured.NestedInt64(msObj.Object, "spec", "replicas")
if !foundReplicas {
    originalReplicas = 1
}

...Defaulting to 1 rather than 0, since the MachineSet must have at least one machine for the test to have found a worker node.


That being said, I think this is a minor issue and shouldn't be blocking for the PR, so /lgtm

@abushkin-redhat

Copy link
Copy Markdown
Collaborator

/lgtm

Anna-Koudelkova and others added 5 commits November 13, 2026 15:05
ValidateMachineSetScaleWithDeletePolicyNewest took status.readyReplicas
as the MachineSet's original size. That field is a transient observation,
and the helper samples it immediately after ApplyRemediationsWithDependencies,
i.e. while MCO may still be rebooting workers, so readyReplicas is often
below spec.replicas.

When that happens the test is both useless and destructive: with
spec.replicas=3 and readyReplicas=2, the "scale up" writes replicas=3,
which is a no-op that satisfies the wait as soon as the rebooting node
comes back, and the "scale down" then writes replicas=2 and permanently
removes a real worker. A single-replica MachineSet whose node is briefly
NotReady reads as readyReplicas=0 and gets scaled to zero.

Take the baseline from spec.replicas, which is the only field that states
the desired size, and treat a missing spec.replicas as an error rather
than defaulting to zero. Gate the whole flow on the MachineSet being
converged (status.replicas and status.readyReplicas both matching
spec.replicas) so an unsettled MachineSet fails fast with a clear message
instead of silently burning the scale-up timeout. Checking status.replicas
as well as readyReplicas also catches a machine that is already being
provisioned or deleted.

The snapshot is now a struct; it had outgrown a five-value return.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cleanup defer is registered before anything scales the MachineSet, so
a failure in setMachineSetDeletePolicyNewest or
verifyMachineSetDeletePolicyNewest -- a 409 conflict is realistic here,
since the object was fetched several Update round-trips earlier -- still
caused the deferred cleanup to write spec.replicas on a MachineSet the
test had never touched.

Track whether the scale-up was attempted and skip the replicas restore
when it was not. The flag is set before scaleMachineSetAndWait rather
than after it, because that call can fail with spec.replicas already
written, and that case does need restoring.

The deletePolicy restore and the label removal still run unconditionally:
the labels are always applied by that point, and rewriting deletePolicy
to the value it already has is harmless.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every write in the machineset scale test read its object well before it
wrote it, and gave up on the first conflict. The worst case was
setMachineSetDeletePolicyNewest: the object it updated was fetched before
labelExistingMachinesForMachineSet ran N sequential Update round-trips
against the same MachineSet's machines, so the resourceVersion was very
likely stale by the time it was written. The machine-API controllers
write to these objects whenever a machine is created, becomes ready, or
goes away, which is continuous during the post-remediation reboots that
immediately precede this test. A conflict there failed the whole suite
for a reason that has nothing to do with compliance.

Add updateWithConflictRetry, which reads, mutates and writes in one place
and re-reads on conflict, and route the deletePolicy set, both scale
writes, the cleanup replicas and deletePolicy restores, and both machine
label updates through it. client-go's retry.RetryOnConflict is not
vendored and client-go v0.30.0 is not in the module cache, so this uses
the same shape (5 attempts, 10ms apart) without a new dependency.

Dropping the passed-in object also removes the unused msKey parameter
from setMachineSetDeletePolicyNewest and the now-unused obj field on
machineSetState, and lets cleanup distinguish a machine that the
scale-down legitimately removed from a real failure to unlabel it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verifyOriginalMachinesPreserved could not fail. It listed machines
immediately after scaleMachineSetAndWait returned, and that returns as
soon as status.readyReplicas is back to the original count -- which the
MachineSet controller satisfies the moment it stamps a deletionTimestamp
on its victim. The Machine object itself survives the whole drain and
deprovision, minutes later. So in the exact regression this test exists
to catch, the controller deleting an older machine instead of the newest,
the victim was still returned by the List and the check passed. It also
passed on a healthy cluster, for the same reason: the assertion returned
nil either way.

Treat a machine with a deletionTimestamp as not preserved, and turn the
check from "nothing went missing" into a positive assertion. Capture the
name of the machine the scale-up added while it is still the only new
one, then require that specific machine to be the one going away and
every original to still be live. The check now polls, because the
controller may not have picked a victim at all when readyReplicas first
drops, and fails fast via backoff.Permanent when an original machine is
the one being deleted.

Machine listing moves into listMachinesForMachineSet, which splits live
from deleting and is shared with the labeling path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Nov 13, 2026

Copy link
Copy Markdown

@Anna-Koudelkova: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-openshift-node-compliance 4a9d915 link false /test e2e-aws-openshift-node-compliance

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants