DIFFUSION

tl;dr We introduce the multimodal conditioning module (MCM), a small modulation network that enables multimodal image synthesis using pretrained diffusion models without any updates to the diffusion model parameters... Results of using MCM to condition Stable Diffusion on new modalities (underlined)... We present multimodal conditioning modules (MCM) for enabling conditional image synthesis using pretrained diffusion models. Previous multimodal synthesis works rely on training networks from scratch or fine-tuning pretrained networks, both of which are computationally expensive for large, state-of-the-art diffusion models. Our method uses pretrained networks but does not require any updates to the diffusion network's parameters. MCM is a small module trained to modulate the diffusion network's predictions during sampling using 2D modalities (e.g., semantic segmentation maps, sketches) that were unseen during the original training of the diffusion model. We show that MCM enables user..