The Stable Signature: Rooting Watermarks in Latent Diffusion Models

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Oct 6 2023

In an article recently submitted to the ArXiv* server, researchers introduced an active strategy combining image watermarking and Latent Diffusion Models (LDMs) to address ethical concerns in generative image modeling. The approach embeds an invisible watermark in generated images for future detection and identification, demonstrating robustness even when images are modified. The method achieved higher accuracy in identifying the source of an image generated from a text prompt, showcasing its potential for responsible deployment of generative models.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

Recent advancements in generative modeling and natural language processing, exemplified by Stable Diffusion, have facilitated the creation and manipulation of highly realistic images, giving rise to creative tools like ControlNet and Instruct- Pixel-to-Pixel (Pix2Pix). While these developments represent significant progress, they also raise concerns about the authenticity of such images. The ability of generative artificial intelligence (AI) to create convincing synthetic images without easy identification poses risks such as deep fakes, impersonation, and copyright infringement.

Past research in image generation has primarily relied on Generative Adversarial Networks (GANs) and, more recently, Transformers and diffusion models. While GANs have maintained their state-of-the-art status, diffusion models have shown significant promise in text-conditional image generation. However, identifying AI-generated or manipulated images remains challenging, particularly in deepfake scenarios. Various detection methods have been explored, including those based on inconsistencies in generated images, but passive forensics approaches have limitations. Watermarking, a more active technique, has gained attention as a potential solution, offering efficient ways to trace and protect against image manipulation, especially when integrated into the generative process.

Methods Used

The Stable Signature method consists of two phases: pre-training the watermark extractor and fine-tuning the LDM decoder. In the pre-training phase, a watermark extractor network, W, is created using Hiding Data in Denoising (HiDDeN), a deep watermarking method. HiDDeN optimizes the parameters of a watermark encoder (WE) and the extractor network (W) to embed k-bit messages robustly into images. 

Following training, the WE becomes inactive and is discarded, leaving only the extractor network W actively employed for watermark extraction from watermarked images. Watermarking involves encoding a message into a cover image, yielding a watermarked image. The watermark extraction process actively derives a soft message from the watermarked image and then actively computes a message loss by actively comparing it to the original message. Notably, the watermarking process is robust to various image transformations.

In the fine-tuning phase, the LDM decoder, D, is fine-tuned to ensure that generated images contain a specified message, m, which can be extracted by W. This fine-tuning process is compatible with various generative tasks, as it only modifies the decoder without affecting the diffusion process. The process involves encoding an image using the LDM encoder, extracting a message using W from the reconstructed image, and computing a message loss to ensure the desired message is present. Additionally, a perceptual loss controls image distortion. The fine-tuning process optimizes the decoder's weights over a few backpropagation steps, ultimately achieving the desired watermarking performance.

To assess the effectiveness of Stable Signature, researchers actively conduct experiments on generative models that they have actively watermarked with 48-bit signature. These experiments use prompts from the Microsoft Common Objects in Context (MSCOCO) dataset, and they assessed performance for detection and identification, with robustness tested against different image transformations. Detection results show that Stable Signature effectively identifies generated images, even after significant modifications, while maintaining a low false positive rate. Identification results demonstrate that the method can accurately attribute generated images to specific users, with a minimal rate of false accusations, even when many users are involved. 

Experimental Results

The experimental results actively demonstrate the effectiveness of Stable Signature across various abundant tasks, image quality assessment, and, in comparison, post-generation watermarking methods. The research covers tasks like text-to-image generation, image editing, super-resolution, and inpainting using popular datasets for evaluation. Performance metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity score (SSIM), Frechet Inception Distance (FID), and bit accuracy are employed to evaluate image quality and watermark robustness against different transformations. The results demonstrate that Stable Signature minimally impacts image generation quality while maintaining robust watermarking capabilities. Comparisons with post-hoc watermarking methods highlight the efficiency of Stable Signature, and the study explores the trade-offs between image quality and robustness through parameter adjustments and the role of an attack simulation layer in the watermark extractor's training.

Tampering Resilience

Actively investigating the resilience of Stable Signature to intentional tampering attacks involves distinguishing between image-level and network-level threats. In image-level attacks, the assessment focuses on the watermark's resistance to removal and embedding attacks, with effectiveness depending on the distortion budget and the attacker's knowledge of the generative model. Meanwhile, network-level attacks involve exploring model purification, where an attacker fine-tunes the model to remove watermarks, and model collusion, where users combine their models to deceive identification. The study reveals how Stable Signature reacts to these malicious actions, providing insights into its robustness in adversarial scenarios.

Conclusion

In summary, this research demonstrates the capability to embed robust and invisible watermarks into images generated by LDM through a simple decoder fine-tuning process. These watermarks serve the purpose of detecting generated images and identifying their creators with high accuracy without affecting the underlying generative process. This work highlights the importance of watermarking as a proactive approach to publicly releasing image-generative models, emphasizing its societal implications. The code for this approach is openly available for reproducibility. While the experiments incurred a notable computational cost, the environmental impact is relatively modest compared to other fields in computer vision, with an estimated carbon footprint of approximately ten tons of Carbon Dioxide Equivalent (CO₂eq).

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.