HyperDreamBooth: Lightning-Fast Personalized Image Generation with AI

By Dr. Sampath LonkaReviewed by Susha Cheriyedath, M.Sc.Jul 18 2023

In a recent paper submitted to the arXiv* server, researchers proposed HyperDreamBooth, a hypernetwork that efficiently generates personalized weights for generative artificial intelligence (AI) from a single person's image. It offers a faster and more efficient solution for personalized image generation while maintaining high-quality results and diverse styles.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

Recent advancements in text-to-image (T2I) personalization have opened up new possibilities for creative applications. Specifically, face personalization allows the generation of diverse images of a specific person in various styles. Notably, DreamBooth stands out for its ability to incorporate new subjects into the model without compromising the model's prior and preserving subject essence, even in vastly different styles like animated cartoons. However, DreamBooth's size and speed limitations hinder its broader impact.

To overcome these shortcomings, the researchers proposed Lightweight DreamBooth (LiDB), a text-to-image model for personalized image generation with a smaller size (around 100KB) achieved by employing low-dimensional weight-space training. Additionally, they introduced a new HyperNetwork architecture that accelerates the customization process, making it 25 times faster while maintaining similar performance to DreamBooth.

Finally, they proposed rank-relaxed finetuning, ensuring higher subject fidelity by approximating high-level subject details in LoRA DreamBooth models. The proposed approach involves modifying the output domain and inserting subject knowledge into the generative model by adjusting network weights.

Related work

Various T2I models, including Imagen, DALL-E2, Stable Diffusion (SD), Muse, and Parti, have demonstrated impressive image generation capabilities based on text prompts. Some T2I models, such as SD and Muse, enable conditioning generation with an image through an encoder network. However, existing text- and image-based conditioning methods lack sufficient subject details.

Personalization of generative models aims to generate subject-specific images in various contexts using a few subject images. Prior approaches, such as pivotal tuning, StyleGAN finetuning, and conditioning GANs, have encountered issues related to poor subject fidelity or limited context diversity. The proposed approach in the current study proposes a hypernetwork-based method to directly predict low-rank network residuals for a given subject, distinct from existing techniques.

Methodology

Preliminaries: In T2I diffusion models, Latent Diffusion Models (LDM) denoise a noise map into an image using a text prompt. The researchers used SD, consisting of an image encoder, decoder, and U-Net denoising network. The DreamBooth model fine-tunes the T2I denoising network for subject-specific images, but this process is slow and memory-intensive. To address this, low-rank adaptation (LoRA) offers a memory-efficient and faster technique by finetuning network weight residuals using low-rank matrices.

Method: The proposed method comprises three core elements: LiDB, HyperNetwork training, and rank-relaxed fast fine-tuning. LiDB minimizes personalized weights while maintaining subject fidelity, editability, and style diversity by introducing a low-dimensional weight space, reducing the model's size significantly. HyperNetwork predicts LiDB low-rank residuals, speeding the personalization process. A transformer decoder is used, iterating predictions to improve results. Rank-relaxed fast fine-tuning captures fine details by employing rank relaxation before LoRA model fine-tuning. This improves subject fidelity compared to fixed-rank weight updates. The proposed method achieves strong personalized results with a fraction of the parameters and fast training, outperforming DreamBooth and LoRA DreamBooth.

Experiments

The researchers implemented HyperDreamBooth on the SD v1.5 diffusion model and predicted LoRa weights for self- and cross-attention layers of the diffusion UNet and the CLIP text encoder. Synthetic face images from the SFHQ dataset were used for visualizations due to privacy concerns, and training utilized 15,000 images from CelebA-HQ. The method demonstrated robust subject personalization with superior or equal performance to state-of-the-art optimization-driven techniques. It achieved high editability, transforming face identities into various domains while preserving the model's style diversity.

Qualitative and quantitative comparisons with Textual Inversion and DreamBooth showed HyperDreamBooth outperformed these methods in most cases. A user study comparing the proposed method's face identity preservation to DreamBooth and Textual Inversion indicated a strong preference for the presented model.

Societal impact

This research aims to provide users with a creative tool for expressive image creation. However, advanced image generation methods can have complex societal implications, inheriting concerns about altering sensitive characteristics and potential biases. The proposed HyperDreamBooth does not introduce new risks, but further research should address bias and harmful content in generative modeling and personalization.

Conclusion

In conclusion, HyperDreamBooth represents a novel approach for rapid and lightweight personalized text-to-image diffusion models. Using a HyperNetwork, it generates LiDB parameters with fast, rank-relaxed finetuning, significantly reducing size and speed compared to DreamBooth. The method successfully produces high-quality, diverse face images with various styles and semantic modifications while preserving subject details and model integrity.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.