Loading .safetensors Lora

Describe the bug

I have downloaded lora from civitai which is in .safetensor format.

When i load it using below code,

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe.unet.load_attn_procs("lora.safetensors")

It throws error : KeyError: 'to_k_lora.down.weight'

File "/workspace/server/tasks.py", line 346, in txt2img
self.pipe.unet.load_attn_procs(embd, use_safetensors=True)
File "/opt/conda/envs/ldm/lib/python3.8/site-packages/diffusers/loaders.py", line 224, in load_attn_procs
rank = value_dict["to_k_lora.down.weight"].shape[0]
KeyError: 'to_k_lora.down.weight'

Reproduction

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe.unet.load_attn_procs("lora.safetensors")

Logs

No response

System Info

Diffusers Version: 0.15.0.dev0

@adhikjoshi

Hey @adhikjoshi,

Thanks for the issue we should indeed try to support also A1111 loading of LoRA tensors soon. cc @sayakpaul here

Kohya-ss/sd-scripts has a nice mechanism for it, but it broke with 0.15, but you can for sure load A1111 LoRA Tensors with the function below on 0.14.0:

def apply_lora(pipe, lora_path, weight:float = 1.0):
    from safetensors.torch import load_file
    from sd-scripts.networks.lora import create_network_from_weights
    import torch
    
    vae = pipe.vae
    text_encoder = pipe.text_encoder
    unet = pipe.unet

    sd = load_file(lora_path)
    lora_network, sd = create_network_from_weights(weight, None, vae, text_encoder, unet, sd)
    lora_network.apply_to(text_encoder, unet)
    lora_network.load_state_dict(sd)
    lora_network.to("cuda", dtype=torch.float16)

but as of 0.15 it fails:

assert lora.lora_name not in names, f"duplicated lora name: {lora.lora_name}"
AssertionError: duplicated lora name: lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_q

@haofanwang

Kohya-ss/sd-scripts has a nice mechanism for it, but it broke with 0.15, but you can for sure load A1111 LoRA Tensors with the function below on 0.14.0:


def apply_lora(pipe, lora_path, weight:float = 1.0):

    from safetensors.torch import load_file

    from sd-scripts.networks.lora import create_network_from_weights

    import torch

    

    vae = pipe.vae

    text_encoder = pipe.text_encoder

    unet = pipe.unet



    sd = load_file(lora_path)

    lora_network, sd = create_network_from_weights(weight, None, vae, text_encoder, unet, sd)

    lora_network.apply_to(text_encoder, unet)

    lora_network.load_state_dict(sd)

    lora_network.to("cuda", dtype=torch.float16)

but as of 0.15 it fails:


assert lora.lora_name not in names, f"duplicated lora name: {lora.lora_name}"

AssertionError: duplicated lora name: lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_q

CC @haofanwang @sayakpaul

Can someone provide LoRA file in the A1111 format? Providing as many relevant details associated to the file as possible would be great too.

Can someone provide LoRA file in the A1111 format? Providing as many relevant details associated to the file as possible would be great too.

I have downloaded offset noise trained lora and uploaded its .safetensor on huggingface

https://huggingface.co/adhikjoshi/epi_noiseoffset

@sayakpaul

@sayakpaul here you go. This Lora was trained using ss-kohya's scripts and works fine in A1111. I can load it on diffusers 0.14.0 with the snippet above using the lora.py from sd-scripts:

caAos-000001.zip

Thanks,

Alejandro.

@patrickvonplaten

Cc: @patrickvonplaten ^

Fix regression introduced in huggingface#2448 (huggingface#2551)

@sayakpaul

Here is function i made from convert_lora_safetensor_to_diffusers.py to load lora on inference time.

import torch
from safetensors.torch import load_file

def load_lora_weights(pipeline, checkpoint_path):
    # load base model
    pipeline.to("cuda")
    LORA_PREFIX_UNET = "lora_unet"
    LORA_PREFIX_TEXT_ENCODER = "lora_te"
    alpha = 0.75
    # load LoRA weight from .safetensors
    state_dict = load_file(checkpoint_path, device="cuda")
    visited = []

    # directly update weight in diffusers model
    for key in state_dict:
        # it is suggested to print out the key, it usually will be something like below
        # "lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight"

        # as we have set the alpha beforehand, so just skip
        if ".alpha" in key or key in visited:
            continue

        if "text" in key:
            layer_infos = key.split(".")[0].split(LORA_PREFIX_TEXT_ENCODER + "_")[-1].split("_")
            curr_layer = pipeline.text_encoder
        else:
            layer_infos = key.split(".")[0].split(LORA_PREFIX_UNET + "_")[-1].split("_")
            curr_layer = pipeline.unet

        # find the target layer
        temp_name = layer_infos.pop(0)
        while len(layer_infos) > -1:
            try:
                curr_layer = curr_layer.__getattr__(temp_name)
                if len(layer_infos) > 0:
                    temp_name = layer_infos.pop(0)
                elif len(layer_infos) == 0:
                    break
            except Exception:
                if len(temp_name) > 0:
                    temp_name += "_" + layer_infos.pop(0)
                else:
                    temp_name = layer_infos.pop(0)

        pair_keys = []
        if "lora_down" in key:
            pair_keys.append(key.replace("lora_down", "lora_up"))
            pair_keys.append(key)
        else:
            pair_keys.append(key)
            pair_keys.append(key.replace("lora_up", "lora_down"))

        # update weight
        if len(state_dict[pair_keys[0]].shape) == 4:
            weight_up = state_dict[pair_keys[0]].squeeze(3).squeeze(2).to(torch.float32)
            weight_down = state_dict[pair_keys[1]].squeeze(3).squeeze(2).to(torch.float32)
            curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down).unsqueeze(2).unsqueeze(3)
        else:
            weight_up = state_dict[pair_keys[0]].to(torch.float32)
            weight_down = state_dict[pair_keys[1]].to(torch.float32)
            curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down)

        # update visited list
        for item in pair_keys:
            visited.append(item)

    return pipeline

Can use it like,

lora_model = lora_models + "/" + opt.lora + ".safetensors"
self.pipe = load_lora_weights(self.pipe, lora_model)

@sayakpaul @patrickvonplaten

Thanks! Do you have the checkpoints with which we could test this?

Can someone provide LoRA file in the A1111 format? Providing as many relevant details associated to the file as possible would be great too.

I have downloaded offset noise trained lora and uploaded its .safetensor on huggingface

https://huggingface.co/adhikjoshi/epi_noiseoffset

This uploaded safetensor lora and others work well

@adhikjoshi

Thanks @adhikjoshi! Getting a lot further with your function but the output is not matching what I would expect. As a first guess, I would think this is the alpha handling as that is hard-coded to 0.75 but the LoRA's I'm using have .alpha keys in them.

How would you like adjust the multiplier?

The same value is used on different layers, right? Say that I have initialized it with 1.0, then I just want to update it like multiplier = 0.75, updating everywhere it's referenced in the network.

Is it something possible to achieve?

I think you would need to fetch the multiplier appropriately from the corresponding state dict accordingly for that. Since the multiplier only concerns the weight modification part, I don't think reinitialization would be required here.

Hello, I'm using version v0.18 of Diffuser, and I managed to load a LoRa .safetensors file from CivitAI. However, I don't notice any changes even when I run lora:test:1.
Do you have any idea what could be causing this issue? Thank you in advance for your help.

With SDXL old lora won't be working.

Any workarounds?

@sonaterai

@sonaterai I think there are some extra keys and corresponding weight parameters for which we don't have support yet. See #3087. It also links to a couple of other similar threads.

With SDXL old lora won't be working.

It's not supposed as the corresponding UNet is different as far as I understand.

@sonaterai

@sonaterai I think there are some extra keys and corresponding weight parameters for which we don't have support yet. See #3087. It also links to a couple of other similar threads.

With SDXL old lora won't be working.

It's not supposed as the corresponding UNet is different as far as I understand.

If i train the Lora base on SDXL by myself, is that anything way to load the new Lora?

If i train the Lora base on SDXL by myself, is that anything way to load the new Lora?

You should be able to. If not, please open a new thread with a reproducible set of instructions.

@sayakpaul

Here is function i made from convert_lora_safetensor_to_diffusers.py to load lora on inference time.

import torch
from safetensors.torch import load_file

def load_lora_weights(pipeline, checkpoint_path):
    # load base model
    pipeline.to("cuda")
    LORA_PREFIX_UNET = "lora_unet"
    LORA_PREFIX_TEXT_ENCODER = "lora_te"
    alpha = 0.75
    # load LoRA weight from .safetensors
    state_dict = load_file(checkpoint_path, device="cuda")
    visited = []

    # directly update weight in diffusers model
    for key in state_dict:
        # it is suggested to print out the key, it usually will be something like below
        # "lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight"

        # as we have set the alpha beforehand, so just skip
        if ".alpha" in key or key in visited:
            continue

        if "text" in key:
            layer_infos = key.split(".")[0].split(LORA_PREFIX_TEXT_ENCODER + "_")[-1].split("_")
            curr_layer = pipeline.text_encoder
        else:
            layer_infos = key.split(".")[0].split(LORA_PREFIX_UNET + "_")[-1].split("_")
            curr_layer = pipeline.unet

        # find the target layer
        temp_name = layer_infos.pop(0)
        while len(layer_infos) > -1:
            try:
                curr_layer = curr_layer.__getattr__(temp_name)
                if len(layer_infos) > 0:
                    temp_name = layer_infos.pop(0)
                elif len(layer_infos) == 0:
                    break
            except Exception:
                if len(temp_name) > 0:
                    temp_name += "_" + layer_infos.pop(0)
                else:
                    temp_name = layer_infos.pop(0)

        pair_keys = []
        if "lora_down" in key:
            pair_keys.append(key.replace("lora_down", "lora_up"))
            pair_keys.append(key)
        else:
            pair_keys.append(key)
            pair_keys.append(key.replace("lora_up", "lora_down"))

        # update weight
        if len(state_dict[pair_keys[0]].shape) == 4:
            weight_up = state_dict[pair_keys[0]].squeeze(3).squeeze(2).to(torch.float32)
            weight_down = state_dict[pair_keys[1]].squeeze(3).squeeze(2).to(torch.float32)
            curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down).unsqueeze(2).unsqueeze(3)
        else:
            weight_up = state_dict[pair_keys[0]].to(torch.float32)
            weight_down = state_dict[pair_keys[1]].to(torch.float32)
            curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down)

        # update visited list
        for item in pair_keys:
            visited.append(item)

    return pipeline

Can use it like,

lora_model = lora_models + "/" + opt.lora + ".safetensors"
self.pipe = load_lora_weights(self.pipe, lora_model)

@sayakpaul @patrickvonplaten

in my case in diffusers0.14.0, it gets wrong :
Traceback (most recent call last):
File "demo_txt2img.py", line 88, in
demo.loadEngines(args.engine_dir, args.onnx_dir, args.onnx_opset,
File "/workspace/demo/Diffusion/stable_diffusion_pipeline.py", line 288, in loadEngines
model = obj.get_model()
File "/workspace/demo/Diffusion/models.py", line 300, in get_model
return load_lora_weights(basic_unet, self.lora, "unet")
File "/workspace/demo/Diffusion/safetensors2bin.py", line 49, in load_lora_weights
temp_name += "_" + layer_infos.pop(0)
IndexError: pop from empty list

can u help me ? thx

Please use the load_lora_weights() API for this (docs). We don't maintain that script, so, sadly cannot offer any support.

Please use the load_lora_weights() API for this (docs). We don't maintain that script, so, sadly cannot offer any support.

however, i want to load lora for my unet and clip, not for the pipeline（Because I reused TensorRT's code）

@sayakpaul

@sayakpaul

I tried the load_lora_weights() but inference results are confuse. load_lora_weights load the text encoder params in the lora?

I tried the load_lora_weights() but inference results are confuse. load_lora_weights load the text encoder params in the lora?

yes , both clip and unet

I am having a similar issue. When I run pipe.load_lora_weights("lora_model_id"), where lora_model_id is the path to my model, I simply get this error:

KeyError: 'lora.down.weight'

Does anyone know how to fix this issue?

Can you open a new issue with a reproducible code snippet?

Navigation Menu

Repository navigation

Loading .safetensors Lora #3064

Describe the bug

Reproduction

Logs

System Info

63 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Loading .safetensors Lora #3064

Description

Describe the bug

Reproduction

Logs

System Info

Activity

patrickvonplaten commented on Apr 12, 2023

alejobrainz commented on Apr 12, 2023

adhikjoshi commented on Apr 12, 2023

sayakpaul commented on Apr 13, 2023

adhikjoshi commented on Apr 13, 2023

alejobrainz commented on Apr 14, 2023

sayakpaul commented on Apr 14, 2023

adhikjoshi commented on Apr 17, 2023

sayakpaul commented on Apr 17, 2023

adhikjoshi commented on Apr 17, 2023

pdoane commented on Apr 17, 2023

63 remaining items

notdanilo commented on Jun 29, 2023

sayakpaul commented on Jun 29, 2023

sonaterai commented on Jul 7, 2023

adhikjoshi commented on Jul 7, 2023

sayakpaul commented on Jul 7, 2023

sanbuphy commented on Jul 8, 2023

sayakpaul commented on Jul 8, 2023

kkwhale7 commented on Sep 1, 2023

sayakpaul commented on Sep 1, 2023

kkwhale7 commented on Sep 1, 2023

kkwhale7 commented on Sep 1, 2023

nullarch commented on Oct 2, 2023

kkwhale7 commented on Oct 2, 2023

alirezaomneky commented on Dec 18, 2023

sayakpaul commented on Dec 19, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions