Skip to content

sd: improve model detection and add support for interleaved image prompts - #2648

Open
wbruna wants to merge 5 commits into
LostRuins:concedo_experimentalfrom
wbruna:kcpp_sd_vlm_images
Open

wbruna wants to merge 5 commits into
LostRuins:concedo_experimentalfrom
wbruna:kcpp_sd_vlm_images

Conversation

@wbruna

@wbruna wbruna commented Nov 15, 2026

Copy link
Copy Markdown

Detects some model types (currently tokenizers, mmprojs and the p-h3 connector) by inspecting the file, and add support for passing images to the VLM ( leejet/stable-diffusion.cpp@5d78740 ).

The improved detection is needed for the image prompts, because a lot of new models can now load mmprojs.

Also includes p-h3 connector detection support.

Moved from, and on top of, #2576 . Moving here the discussion too:

  • tags <image:N> map to the extra images array; they are by default removed from the reference images passed to sd.cpp
  • tags <image:+N> mean "image for both the VLM and the reference image list"
  • img2img with VLM image tags is supported, although the stable-ui currently doesn't support it (partial denoise with the VLM guiding it could be a nice use case)

A few arguably questionable choices 🙂 :

  • indexing is one-based, but we try to read the user's mind: if they use a tag <image:0>, the index becomes zero-based, instead of the whole thing failing. Happy to drop it, if you think it's too magical.
  • on a model with no vision support, sd.cpp simply fails the generation if there are any image tags. I've added code to replace <image:N> with imageN, to keep the prompt consistent, and give a chance to an edit model to understand it. Again, no hard feelings if you prefer aborting the gen.
  • the '+' indicator was the simplest way I could think of to communicate which images should still be treated as normal references. Proper support would demand tagging the references themselves, and would need UI support and API changes... I don't care much about the exact syntax; but it should fit into "image:$SOMETHING" (we could have our own syntax for everything, but...)

@wbruna
wbruna force-pushed the kcpp_sd_vlm_images branch 2 times, most recently from 877e9a6 to eba0eec Compare November 16, 2026 22:03
@wbruna
wbruna force-pushed the kcpp_sd_vlm_images branch from eba0eec to b0d316e Compare November 17, 2026 22:31
@LostRuins

Copy link
Copy Markdown
Owner

Will review and merge this soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants