Skip to content

dreambooth lora: text encoder monkey-patching not working #3445

Closed
Closed
dreambooth lora: text encoder monkey-patching not working#3445
Assignees
sayakpaul
Labels
bugSomething isn't workingstaleIssues that haven't received updates

Description

@rvorias
rvorias
opened on May 16, 2023

Describe the bug

In the dreambooth lora script, the lora layers are monkey patched into the text encoder as the text encoder comes from transformers. However, this has some undesirable results:

image

This is before any weight is updated. Vertically, the images share the same exact seed.

I suspect these lines of code:

diffusers/src/diffusers/loaders.py

Lines 949 to 950 in 29b1325

def new_forward(x):
return old_forward(x) + lora_layer(x)

Commenting out the lora term produces the same distortions. This means that wrapping the original ref to the forward function and injecting it via monkey patching produces extra undesirable results.

My current guess is that this is some pass-by-ref issue.

Reproduction

train_text_encoder: True and check images at epoch=0 before training.

Logs

No response

System Info

Python 3.8.10

accelerate==0.19.0
diffusers @ git+https://github.com/huggingface/diffusers.git@7a32b6beeb0cfdefed645253dce23d9b0a78597f
transformers==4.29.1

Activity

  1. rvorias
    added
    bugSomething isn't working
    on May 16, 2023

  2. rvorias commented on May 16, 2023

    @rvorias
    rvorias
    on May 16, 2023
    Author

    This seems to alleviate the ref issue

    def create_new_forward(old_forward):
    def new_forward(x):
        return old_forward(x)
    return new_forward

# Monkey-patch.
module.forward = create_new_forward(old_forward)

    So finally:

    ...
old_forward = module.forward

def create_new_forward(old_forward):
    def new_forward(x):
        return old_forward(x) + lora_layer(x)
    return new_forward

# Monkey-patch.
module.forward = create_new_forward(old_forward)

  3. patrickvonplaten commented on May 16, 2023

    @patrickvonplaten
    patrickvonplaten
    on May 16, 2023
    Contributor

    cc @sayakpaul here

  4. patrickvonplaten
    assigned
    sayakpaul
    on May 16, 2023

  5. Mr-Philo commented on May 17, 2023

    @Mr-Philo
    Mr-Philo
    on May 17, 2023
    Contributor

    Same problems here! Adding --train_text_encoder will lead to broken text encoder: the generated images won't follow the input text:

    image

    When training after 2500steps, the model will fit the training images but still ignores the input prompt, llike here "the sunflowers":

    image

    When disable --train_text_encoder, everything goes smoothly.

    Experiment Setting

    transformers==4.28.1
accelerate==0.19.0
diffusers=0.17.0.dev0 (latest)

  6. sayakpaul commented on May 17, 2023

    @sayakpaul
    sayakpaul
    on May 17, 2023
    Member

    Being worked in #3437. Thanks for your patience.

  7. github-actions commented on Jun 15, 2023

    @github-actions
    github-actionsBot
    on Jun 15, 2023
    Contributor

    This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

    Please note that issues that do not follow the contributing guidelines are likely to be ignored.

  8. github-actions
    added
    staleIssues that haven't received updates
    on Jun 15, 2023

  9. patrickvonplaten commented on Jun 16, 2023

    @patrickvonplaten
    patrickvonplaten
    on Jun 16, 2023
    Contributor

    Think we can close this as it's solved, please leave a message if it's not solved for you

  10. patrickvonplaten
    closed this as completedon Jun 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingstaleIssues that haven't received updates

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions