[IP-Adapter] adding IP adapter support to all ControlNet and T2I pipelines

we recently added IP-adapter support to many of our pipelines including all the text2img, img2img and inpaint pipelines, as well as the text2img ControlNet pipeline.

anyone interested in adding it to rest of the ControlNet pipelines and T2I pipelines? You can check out this PR to see how it should be added #5713

@yiyixuxu

Hi @yiyixuxu,

I'd like to give it a try and add support for the StableDiffusionControlNetInpaintPipeline

@juancopi81

Awesome! @juancopi81 Will you be working on controlnets or other pipelines as well? If not, I'd be happy to take it up for txt2img :)

Sure! 😃 go ahead, I'll be working for now only in this pipeline: StableDiffusionControlNetInpaintPipeline

@yiyixuxu

Hey @yiyixuxu I'll also look into it :)

@juancopi81

Hey @juancopi81 @a-r-r-o-w is it okay if I take up for remaining one(img2img)? #self-assign

@charchit7

Hi @charchit7, for me it is ok, I took the StableDiffusionControlNetInpaintPipeline and I already created the PR, so in my case I am not working for the moment in any other pipeline.

@a-r-r-o-w

@a-r-r-o-w I am almost done with img2img :)

@yiyixuxu

@yiyixuxu I am interested in working on this issue and I am wondering if there is any more work I can contribute to?

@dverdu-freepik

do you mean you need to set different adapter_conditioning_scale at different steps? can you explain why do you need to do this?

@dverdu-freepik I think it makes sense too!
do you want to open a PR for it? we can treat it similarly to how we treat lora_scale

I've started a draft PR to add support to StableDiffusionXLControlNetInpaintPipeline: #6605
But I don't really know what I'm doing. I basically followed the changes in #5887 to apply by best guesses. Any help in pushing this forward will be appreciated!

How to unload ip adapter?

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")

I used to .load_ip_adapter, but what if laterwards I want to unload the ipadapter?

@wilfrediscoming

@wilfrediscoming
we have a unload_ip_adapter method

diffusers/src/diffusers/loaders/ip_adapter.py

Line 229 in d5f444d

def unload_ip_adapter(self):

But I got an error after using the method..

inpaint.unload_ip_adapter()

prompt = "on the rock"
neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"

images = inpaint(
    prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
    negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",
    num_inference_steps=25,
    image=image_blank,
    mask_image=image_mask,
    control_image=depth_image,
    # ip_adapter_image=image_ref,
    # guidance_scale=15,
    strength=1.0,
    controlnet_conditioning_scale=1.0,
    num_images_per_prompt=4,
    # clip_skip=1,
)


idx = 0

for each in images.images:
    each.save('/content/semi-final.jpg')

    image_product = Image.open("/content/product-512.png")
    each.paste(image_product, (x,y), image_product)

    each.save(f'/content/final{idx}.jpg')
    idx += 1

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-8-54f9dce33df7>](https://localhost:8080/#) in <cell line: 8>()
      6 neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"
      7 
----> 8 images = inpaint(
      9     prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
     10     negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",

2 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py](https://localhost:8080/#) in run_safety_checker(self, image, device, dtype)
    663             else:
    664                 feature_extractor_input = self.image_processor.numpy_to_pil(image)
--> 665             safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device)
    666             image, has_nsfw_concept = self.safety_checker(
    667                 images=image, clip_input=safety_checker_input.pixel_values.to(dtype)

TypeError: 'NoneType' object is not callable

I believe this can be marked as complete since it has been sufficiently addressed. Thanks for the great work everyone!

Navigation Menu

Repository navigation

[IP-Adapter] adding IP adapter support to all ControlNet and T2I pipelines #5884

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[IP-Adapter] adding IP adapter support to all ControlNet and T2I pipelines #5884

Description

Activity

juancopi81 commented on Nov 21, 2023

a-r-r-o-w commented on Nov 21, 2023

juancopi81 commented on Nov 21, 2023

charchit7 commented on Nov 22, 2023

charchit7 commented on Nov 22, 2023

juancopi81 commented on Nov 22, 2023

charchit7 commented on Nov 22, 2023

jenniferzxm commented on Nov 26, 2023

yiyixuxu commented on Nov 26, 2023

yiyixuxu commented on Nov 27, 2023

atesgoral commented on Jan 16, 2024

wilfrediscoming commented on Feb 20, 2024

yiyixuxu commented on Feb 22, 2024

wilfrediscoming commented on Mar 5, 2024

a-r-r-o-w commented on Nov 19, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions