Difference between revisions of "Diffusion"
m |
m (→ControlNet) |
||
| Line 24: | Line 24: | ||
= ControlNet = | = ControlNet = | ||
| − | |||
* [https://arxiv.org/abs/2302.05543 Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala] | * [https://arxiv.org/abs/2302.05543 Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala] | ||
| + | * [https://github.com/lllyasviel/ControlNet ControlNet | llyasviel] | ||
| + | * [https://medium.com/intuitionmachine/deep-learning-modularity-and-language-models-bd726c5e3b58 Deep Learning Modularity and Language Models | Carlos E. Perez - Medium] | ||
We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications. [https://arxiv.org/abs/2302.05543 Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala] | We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications. [https://arxiv.org/abs/2302.05543 Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala] | ||
https://github.com/lllyasviel/ControlNet/raw/main/github_page/he.png | https://github.com/lllyasviel/ControlNet/raw/main/github_page/he.png | ||
Revision as of 20:48, 16 February 2023
YouTube search... ...Google search
- Markov Model (Chain, Discrete Time, Continuous Time, Hidden)
- Stable Diffusion | Stable Diffusion ... a latent text-to-image diffusion model capable of generating photo-realistic images given any text input
- DALL-E 2 | OpenAI
Diffusion Models are generative models, meaning that they are used to generate data similar to the data on which they are trained. Fundamentally, Diffusion Models work by destroying training data through the successive addition of Gaussian noise, and then learning to recover the data by reversing this noising process. | Ryan O'Connor - AssmblyAI
ControlNet
- Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala
- ControlNet | llyasviel
- Deep Learning Modularity and Language Models | Carlos E. Perez - Medium
We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications. Adding Conditional Control to Text-to-Image Diffusion Models | Lvmin Zhang, Maneesh Agrawala
