Efficient Text-to-Video Generation and Editing Using Simple Diffusion Adapter

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Aug 23 2023

In an article recently submitted to the ArXiv* server, researchers addressed the success of Text-to-Image (T2I) AI technology and the lag in Text-to-Video (T2V) development by introducing a "Simple Diffusion Adapter" (SimDA) as an efficient way to adapt a strong T2I model for T2V, using lightweight spatial and temporal adapters. This method also introduced Latent-Shift Attention (LSA) for temporal consistency and included video super-resolution capabilities. SimDA demonstrated potential for one-shot video editing, reducing training effort while achieving effective adaptation with minimal parameter tuning.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

Image generation is a prominent aspect of recent artificial intelligence (AI) advancements, with applications spanning computer graphics, art, and medicine. Although techniques like generative adversarial networks (GANs), auto-regressive transformers, and diffusion models are prevalent,  video generation lags due to data scarcity and high costs. While some T2V methods fine-tune T2I models for video, challenges persist due to parameters. In Natural Language Processing (NLP) and computer vision, efficient fine-tuning strategies have emerged to manage the cost of large models.

Related work

Previous research in T2V Generation focused on using GAN for specific video domains, with challenges like data scarcity and modeling complexity hindering broader development. Leveraging knowledge from T2I models, recent efforts like CogVideo, Make-A-Video, and others aimed to enhance T2V. Text-guided video editing techniques emerged for manipulating images and videos using text-based input. Parameter-efficient transfer learning approaches in NLP and computer vision paved the way for adapting large models with reduced computation costs. The introduction of the Temporal Shift Module (TSM) facilitated the incorporation of temporal cues, and the LSA further refined this by applying it to video generation tasks, ensuring temporal consistency and improved quality.

Proposed method

In this present paper, the SimDA is introduced as a parameter-efficient video diffusion model that enhances video generation by fine-tuning the large T2I model, specifically the Stable Diffusion. Adding just 0.02% parameters compared to the T2I model significantly improves performance. During training, the original T2I model is kept unchanged, and only the newly added modules are modified.

Moreover, the LSA mechanism is proposed to substitute the initial spatial attention, thereby enhancing temporal modeling and consistency without introducing new parameters. The model requires less than 8GB of Graphics Processing Unit (GPU) memory for training at 16 × 256 × 256 resolution, achieving a remarkable 39× inference speedup compared to the autoregressive approach CogVideo.

Furthermore, an adaptation of an image super-resolution framework enables the generation of high-definition videos at 1024 × 1024 resolution, and the model smoothly extends to diffusion-based video editing, resulting in a three times faster training speed with comparable results. In summary, the contributions include demonstrating the effectiveness of minimal parameter tuning in transitioning from image to video diffusion, leveraging lightweight adapters and LSA for improved temporal modeling, enabling text-instructed video enhancement and editing, and significantly reducing training costs and inference time while still maintaining competitive with other methods.

Experimental results

The T2V method consists of two-stage models: one predicting 256×256 video frames with a latent size of 32×32, and another performing 4× up-sampling to reach 1024×1024 resolution. The general T2V model is trained on the WebVid-10M dataset following, with evaluations reported using Contrastive Language-Image Pretraining (CLIP) score and Frechet Video Distance (FVD) score on Microsoft Research Video-to-Text (MSR-VTT) and WebVid. The comparison includes parameter scale, inference speed, and a user study involving VideoFusion, Latent video diffusion models (LVDM), Video diffusion models (VDM), and Latent-shift. SimDA performs well in both T2V generation and text-guided video editing, showcasing parameter efficiency, high-quality results, and user preference. Ablation studies validate module effectiveness: Temporal Adapter, Spatial Adapter, Attention Adapter, Feedforward Neural Network (FFN) Adapter, and LSA.

Contributions of the paper

The key contributions of this study can be summarized as follows:

1. Parameter-Efficient Video Diffusion Model: The paper introduces a SimDA for T2V generation. SimDA fine-tunes a large T2I model for video generation with only a small increase in parameters, resulting in improved video generation efficiency.
2. LSA: The paper proposes using LSA to replace traditional spatial attention. LSA enhances temporal modeling capabilities and maintains consistency in video generation without introducing additional parameters.
3. Video Super-Resolution and Editing: The SimDA model is extended to support video super-resolution and editing tasks. This versatility allows the model to generate high-definition videos and facilitate faster training for diffusion-based video editing.
4. Comprehensive Evaluation: The proposed method is thoroughly evaluated using various benchmarks and metrics. The paper demonstrates superior performance in CLIP scores, FVD scores, and user preference compared to existing methods for T2V generation and editing.
5. Ablation Studies: The paper conducts ablation studies to analyze the effectiveness of different components of the proposed model, such as Temporal Adapter, Spatial Adapter, Attention Adapter, and LSA.

Conclusion

In summary, the paper presents SimDA, an efficient video diffusion model for text-guided generation and editing. Using lightweight spatial and temporal adapters, the approach successfully transfers spatial information and captures temporal relationships with minimal new parameters. Experimental results demonstrate swift training and inference speeds while maintaining competitive generation and editing outcomes. This work establishes the first parameter-efficient video diffusion method, serving as a valuable T2V fine-tuning baseline and paving the way for future research.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.