Repository navigation
chore(tests): add scheduled sweeper for leaked e2e infrastructure - #5599
Merged
Merged
Conversation
Add a nightly GitHub Actions workflow that deletes stale e2e CloudFormation stacks from the test account. Stacks become eligible 12 hours after creation when they carry the e2e service tag or an LmiShared-* name. Deletion runs in two dependency-ordered passes (function stacks before shared capacity providers) with bounded concurrency and no force deletion; discovery must complete before anything is deleted. Unresolved stacks open or update a single sanitized incident issue and fail the run; a later clean sweep closes it. Manual dispatch defaults to a dry run that only writes a job summary. Also apply the e2e service tag to caller-supplied TestStack stacks so the tag-based selector covers the layer publisher stack. Refs #5519
Enforce a 50-minute wall-clock budget inside the sweeper so the report is always emitted before the 60-minute job cap kills the process; remaining stacks are reported unresolved instead of fabricating a discovery failure. Treat REVIEW_IN_PROGRESS as deletable: it is a terminal status holding no resources, and skipping it would fail every nightly run indefinitely. Make the dispatch dry-run computation fail toward dry run, rename the input to snake_case, add a workflow concurrency group, surface dry-run sweeper errors, and drop the redundant build step. In the notification script, consume report.ok, cap issue-body sections to avoid the 65k limit, guard the label-creation race, and share the markdown formatter with the job summary. Also align JSDoc and unit tests with the repo rules and pin the concurrency limit and time budget with mutation-verified tests. Refs #5519
Create the incident issue without managing any labels and deduplicate by exact title instead of a label lookup. Drop the auto-close path: a clean run does nothing and the incident stays open until a maintainer closes it. Refs #5519
Describe purpose, triggers, failure behavior, and authentication at the top of the workflow, matching the header style of make-release.yml. Refs #5519
svozza
reviewed
Aug 31, 2026
Use the SDK paginateDescribeStacks paginator instead of a hand-rolled NextToken loop, inline the markUnresolved helper at its call sites, and replace the duck-typed client cast in tests with aws-sdk-client-mock so the paginator receives a real CloudFormationClient instance. Refs #5519
dreamorosi
marked this pull request as ready for review
August 31, 2026 14:50
svozza
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a nightly GitHub Actions workflow that removes leaked e2e CloudFormation stacks from the test account, as a final safety net for the hourly-billed LMI capacity provider infrastructure introduced by #5465. Stacks become eligible 12 hours after creation when they carry the
Service=Powertools-for-AWS-e2e-teststag or anLmiShared-*name.Changes
sweepStaleStackstopackages/testing(newe2e:sweepscript): paginated discovery viaDescribeStacks(root stacks only, 12-hour age gate, exact tag orLmiShared-prefix), all-or-nothing discovery, two dependency-ordered deletion passes (function stacks before shared capacity providers), concurrency capped at 5, 20-minute per-stack waits by stack id, no force deletionREVIEW_IN_PROGRESSis terminal and holds no resources, so those are deletedsweep-stale-e2e-stacks.yml: nightly cron plus manual dispatch defaulting to dry run (discovery only, job summary, no issue); reuses the existinge2e-testsOIDC environment; workflow concurrency group prevents overlapping sweepsreport_e2e_sweep.js: on failed real runs, creates or comments on a single incident issue deduplicated by exact title (no labels managed, no auto-close); issue bodies are truncated below GitHub's size limit; the same formatter renders the job summaryTestStackstacks so the tag-based selector also covers the layer publisher stack; drop the now-redundant manual tag insharedCapacityProviderStackIssue number: closes #5519
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.