Repository navigation
The finetuned SDXL shows bad performance on Pokeman #4827
Description
Hello, I just git clone the repo and conduct the training along the guidance here https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/README_sdxl.md where I finetuned the SDXL on Pokeman dataset with script train_text_to_image_sdxl.py. However, the generated images are bad-looking. For example,
A beautiful girl.
A pokemon with green eyes and red legs.
The results are really bad-looking.
Activity
bghira commentedon Aug 29, 2023
what parameters?
fiona-lxd commentedon Aug 29, 2023
The default one here https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/README_sdxl.md . I changed nothing. Is there any parameters that I should be cautious about?
bghira commentedon Aug 29, 2023
let's review.
I am hoping you used madebyollin's VAE.
This is harmless.
Training at a resolution of 512x512 can be destructive. I had some significant damage when training at 256x256 using the default parameters of the Pix2Pix training script. However, training at 768x768 seems to be really beneficial.
20% caption dropout is really high, @sayakpaul was that intentional? .1 is the value in use by SAI.
in fact, it might be best to use zero dropout for the pokemon example, maybe?
A batch size of 1 is not great with AdamW8Bit. It might be all that you can manage with your VRAM pool. But, could you try higher values?
This seems really really high for that small of a training set.
This is fine.
@sayakpaul are you sure you wanted 1e-06 for this? that seems very high.
I would test with
4e-7especially on a batch size of 1.This could be a bad idea.
bf16is a safer choice for numeric stability, especially given that we use OpenCLIP G here.Try validating every epoch.
Checkpoint more frequently.
I don't see
--use_emain here, which increases VRAM use but majorly improves training results.fiona-lxd commentedon Aug 29, 2023
Thank you a lot for the reply and the suggestion. I have started the training again with the hyper-parameters below (vae is madebyollin's VAE):
--enable_xformers_memory_efficient_attention
--resolution=768 \ # it was 512 before
--center_crop
--random_flip
--proportion_empty_prompts=0. \ # it was 0.2 before
--train_batch_size=1
--gradient_accumulation_steps=1 \ # I used 4 A100, therefore, this can be reduced from 4 to 1
--gradient_checkpointing
--max_train_steps=2500 \ # Since we use 4 A100, which means batch-size=4, therefore, the train-step is reduced to 1/4. Actually, I have viewed all the generated images every 500 step, but they are all bad-looking.
--learning_rate=4e-07 \ # Changed from 1e-6
--lr_scheduler=constant
--lr_warmup_steps=0
--mixed_precision=bf16 \ # changed from fp16
--use_ema \ # Added new here
--validation_prompt=a_cute_Sundar_Pichai_creature
--validation_epochs 1
--checkpointing_steps=500
I will update this result once the experiment is done.
bghira commentedon Aug 29, 2023
you want to use as high of a batch size as you can. for your 768px training you should be able to achieve a batch size of 10. using gradient accumulation is good too. leave that as 4.
with each GPU at effective batch size of 10 * 4, with 4 of them, you have effective bsz of 160.
bghira commentedon Aug 29, 2023
FWIW, within 500 steps at 1e-6 without EMA the model is obliterated. this is why you had bad results.
fiona-lxd commentedon Aug 30, 2023
Thank you for the reminder. I'm still waiting for the computational resources. I will update this one I got the results. BTW, do you have any suggestions on finetuning the SDXL on 10 0000 images? Is the hyper-parameter same with the one trained for Pokeman?
fiona-lxd commentedon Aug 30, 2023
OK, here is the result along the training steps. (0, 100, 500, 1000, 1500, 2000 ... , the validation prompt is a_cute_Sundar_Pichai_creature). The batch size is 1 (I cannot install the 8bit adam successfully) and the gradient accumulation is 4. I used 4 GPUs. Therefore, the effective batchsize is 16. I used the fp16 instead of bf16. I found this code has error when using bf16. Specifically, in accelerator.backward(loss), the error "RuntimeError: Expected tensor for argument #1 'grad_output' to have the same type as tensor for argument #2 'weight'; but type CUDABFloat16Type does not equal torch.cuda.HalfTensor (while checking arguments for cudnn_convolution_backward_input)" occurred. Though I have put both the unet and all the inputs to weight_dtype, this cannot resolve the issue.
Anyway, based on the above settings, the results become better than before. But it seems that the model didn't learn to draw the pokeman style. I don't think the model is learning what we designed. Should I degrade the learning rate?
bghira commentedon Aug 30, 2023
is that with zero caption dropout? in that case it seems that the value is needed to be non-zero.this should probably be put back around 20% to ensure style transfer.
fiona-lxd commentedon Aug 30, 2023
Ok, I will try proportion_empty_prompts=0.2.
fiona-lxd commentedon Aug 31, 2023
OK, here are the results. The left is SD1.5 finetuned on Pokeman under the guidance of https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/README.md. The model has learned to generate images in Pokeman's style. In contrast, the middle (proportion_empty_prompts=0.0) the right one (proportion_empty_prompts0.2) are generated by finetuning the SDXL. They all failed.
George0726 commentedon Sep 1, 2023
I confront the same issue. It seems that SDXL is really hard to fine-tune.
github-actions commentedon Oct 18, 2023
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
markrmiller commentedon Nov 26, 2023
I've always found diffusers to have very poor performance regarding SDXL. But I know SDXL is not difficult to fine-tune because I do it all the time kohya sd-scripts.
I never use Diffusers, except for with a Dreambooth inpainting training script I have, which seems to work okay. Outside that, training has never seemed right.
With SD 1.5, when using Lora, you could typically use a larger training rate with no problem. Here, results would go bad fairly quickly. And if you backed off enough to prevent that, the results were poor at picking up the training target, almost like the learning rate was too low, but you couldn't raise it. But then, if you just trained a long time, the images went wrong before they started picking up the training data well - even with 30,000 plus images to help with overfitting.
Beyond picking up the training data, the results just didn't seem reasonable in general. If you output some early validation images, they were good, and then they just got worse every time you output more. Dreambooth style regulation wasn't much help.
So, outside my inpainting script, I've considered diffusers unusable for SDXL training.
I've been looking at it again, though. The sd-scripts repo is annoying to hack on outside of the training loop, and I've got some things I'm playing with - a dataloader that allows mixed-sized images and different latent caching strategies, but that works better than sd-scripts for some dataset sizes I'm playing around with AlignProp reward feedback, and a couple of other things. But this training issue stands in the way of actually using this code now.
The sd-scripts repo has a lot of custom action going on, so it's not entirely straightforward to work out what's going wrong here. I just played around a little, though, and I think part of it may be related to the v prediction. snr_gamma could be related in this case as well; the poor performance pre-dates it, but I've been using it as I do with sd-scripts, and diffusers results seem to be even poorer than they were months ago.
So, initially, I've swapped v-prediction and snr_gamma to how sd-scripts does them. In my test case, which performs worse than it used to, there is a big improvement. It's not likely the only issue, results are not great on the whole, but in this test case, I train dreambooth style on a subject and prompt for that subject in a space suit. Typically, or always in my case, you will get a NASA-style space suit initially, with a glass windowed helmet you can't see through. In typical, normal training, that will change to glass that you can see the subject through and then either an open helmet or no helmet. With the current diffusers script, I get the space suit where you can't see the subject round after round after round. When I swap v-prediction and snr_gamma to the sd-scripts versions, I can see the subject's face within a round or two.
bghira commentedon Nov 26, 2023
well, SDXL doesn't use v-prediction.