Repository navigation
(SDXL) load lora error #4061
Description
Describe the bug
pipe.load_lora_weights(".", model_path)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 137, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLPipeline' object has no attribute 'load_lora_weights'
Reproduction
pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
pipe.scheduler.config, use_karras_sigmas=True
)
model_path = "./models/last-000010.safetensors"
pipe.load_lora_weights(".", model_path)
Logs
No response
System Info
Name: diffusers
Version: 0.18.2
ubuntu, python3.10.8
Who can help?
No response
Activity
patrickvonplaten commentedon Jul 12, 2023
Here we go with SDXL and Loras haha, @zbulrush where did you take the LoRA from / how did you train it?
patrickvonplaten commentedon Jul 12, 2023
cc @sayakpaul
zbulrush commentedon Jul 13, 2023
I was trained using the latest version of kohya_ss. The base mode is lsdxl, and it can work well in comfyui.
I want to do more custom development. If necessary, I can provide the LoRa file.
sayakpaul commentedon Jul 13, 2023
If you install
diffusersfrom source (pip install git+https://github.com/huggingface/diffusers) you should have access toload_lora_weights()method from the SDXL pipeline.But please be aware of the following:
#3725 (comment)
If you load your LoRA file, I am almost certain that you will notice that there are keys and parameters not yet supported by Diffusers. We're working on it here: #3756.
zbulrush commentedon Jul 13, 2023
pipe.load_lora_weights(".", "last-000010.safetensors")
TypeError: LoraLoaderMixin.load_lora_weights() takes 2 positional arguments but 3 were given
sayakpaul commentedon Jul 13, 2023
Could you try with
?
zbulrush commentedon Jul 13, 2023
pipe.load_lora_weights(".", weight_name ="last-000010.safetensors")
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 827, in load_lora_weights
self.load_lora_into_unet(state_dict, network_alpha=network_alpha, unet=self.unet)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 1002, in load_lora_into_unet
unet.load_attn_procs(state_dict)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 321, in load_attn_procs
rank = value_dict["to_k_lora.down.weight"].shape[0]
KeyError: 'to_k_lora.down.weight'
Then I tried another model(https://civitai.com/models/106582/aogamisdxl), and it returned the following error. Both of these models can work on ComfyUI.
pipe.load_lora_weights(".", weight_name ="aogami_v4_12800_style.safetensors")
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 827, in load_lora_weights
self.load_lora_into_unet(state_dict, network_alpha=network_alpha, unet=self.unet)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 995, in load_lora_into_unet
unet.load_attn_procs(unet_lora_state_dict, network_alpha=network_alpha)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/loaders.py", line 326, in load_attn_procs
attn_processor = getattr(attn_processor, sub_key)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 186, in getattr
return super().getattr(name)
File "/root/autodl-tmp/tools/demo/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'UNet2DConditionModel' object has no attribute 'input'
sayakpaul commentedon Jul 13, 2023
Then it seems to be a new format altogether. So, I would like to kindly draw your attention to my comment here: #3725 (comment).
There are numerous LoRA formats in the community, and it's really painful for us to attempt to support all of them.
zbulrush commentedon Jul 13, 2023
enh, I think the lora based on sdxl could be a whole new topic, The structure is different from the previous.
looking forward to update
adhikjoshi commentedon Jul 14, 2023
Can we have some list of supported lora formats?
It's getting confusing now, there's lora, lorcois, loha, lora without text encoder and list goes on.
Or maybe a function to find out if given lora is which format? Even if it doesn't support
isidentical commentedon Jul 17, 2023
This would be super useful for real world diffusers integrations. We are currently building a generic (any base model / checkpoint) SD text to image API with support for any loras that can be uploaded on top of existing diffusers pipelines and utilities. But unfortunately there is no easy way for us to know whether a LoRA is going to work, and if it does with what precision (e.g. unloaded keys).
For a very naive version of this, we started writing an internal script but it is mainly heruistic-based for the examples we saw out in the wild and might not be %100 accurate (e.g. it matches every Conv LoRA to LyCORIS): https://gist.github.com/isidentical/b9133961df0a72109b1bcbcece1a25b7.
Since there is no standardized metadata, it is going to be super hard to keep track of it without a centralized library like diffusers providing the detection support. I'd really love a
guess_lora_type()function or something like that (or better warnings/errors when LoRAs can not be loaded due to incompatibility).sayakpaul commentedon Jul 18, 2023
Would love a contribution on this. I think it should be possible to base its logic from
diffusers/src/diffusers/loaders.py
Line 965 in 8b18cd8
Completely empathize with this and something we're also struggling with unfortunately :(
github-actions commentedon Aug 11, 2023
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.