Instant 3D Texturing with Meta 3D TextureGen

By Muhammad OsamaReviewed by Susha Cheriyedath, M.Sc.Jul 18 2024

In an article recently posted to the Meta Research website, researchers introduced "Meta 3D TextureGen," a novel method for creating realistic and diverse textures for three-dimensional (3D) objects from text descriptions. This technique uses two sequential neural networks that work in image and ultraviolet (UV) space to produce high-quality, globally consistent textures in less than 20 seconds.

Background

Texture generation is crucial in 3D content creation as it controls the appearance of 3D objects and enhances their expressiveness and realism. However, manually creating textures is difficult, time-consuming, and requires specific skills. Therefore, automated methods for generating textures from natural language descriptions are needed for greater intuitiveness and flexibility.

Previous methods for text-driven texture generation have used optimization-based approaches, generative adversarial networks (GANs), and diffusion models. These methods have several limitations, such as a lack of global consistency, semantic alignment, text fidelity, and slow inference speed. For example, some methods suffer from the “Janus effect,” where multiple instances of a feature, like a face or an eye, appear in different places on the object. Additionally, most methods rely on large-scale image or video datasets for pre-training, which are not easily available for 3D generation.

About the Research

In this study, the authors proposed Meta 3D TextureGen for texturing 3D objects from text descriptions, addressing the gaps of previous methods and achieving state-of-the-art results in quality and speed. The process has two main stages: the first stage operates in image space, using text descriptions and 3D shape features to produce renders of the textured shape from multiple views/scenes.

The second stage works in UV space, using a weighted backprojection of the first stage output and 3D shape features to generate a complete UV texture map that is consistent across different views and matches with the given text prompt. An optional third stage enhances texture quality and increases resolution by a factor of four using a patch-based multidiffusion network.

The study uses a diffusion-based neural network for both stages, fine-tuned from a pre-trained image generator. Diffusion models transform data into Gaussian noise and reverse the process through denoising steps. The researchers introduced a texture enhancement network that extends the MultiDiffusion approach from one-dimensional (1D) to two-dimensional (2D) image-patch overlaps. They also made several novel contributions to improve models' performance, such as using position and normal renders for conditioning, generating all texture views jointly, and employing a weighted blending technique.

Furthermore, the authors evaluated their method on a dataset of 26k textured 3D objects from an in-house collection. Then, they compared it with several state-of-the-art methods, including TEXTure, Text2tex, SyncMVD, Paint3D, and Meshy 3.0. They used numerical metrics and user studies to measure the quality, consistency, and text alignment of the generated textures. They also measured the inference time and the resolution of the output textures.

Research Findings

The outcomes showed that Meta 3D TextureGen outperformed all baselines in quality, speed, consistency, and text alignment. In the user study, it was preferred for a better representation of the prompt and fewer artifacts. According to numerical metrics, it achieved the lowest Frechet Inception Distance (FID) and Kernel Inception Distance (KID) scores, indicating better visual fidelity. It was also the fastest method, taking only 19 seconds to generate a texture map, compared to 66-287 seconds for the baselines. Additionally, Meta 3D TextureGen produced 4K pixel resolution textures, while the baselines were limited to 1K pixels.

The generated textures were realistic, diverse, and faithful to the text prompts, without artifacts like the Janus effect or seams. The method handled various shapes and prompts, ranging from realistic to stylized, and produced high-resolution textures up to 4k pixels. It was fast, requiring only a single forward pass over two diffusion processes, generating a texture in less than 20 seconds. The authors demonstrated that diffusion models are well-suited for texture generation, as they can capture complex and diverse distributions and be easily conditioned on multiple modalities like text, images, and UV maps.

Applications

The proposed method can be applied to various domains that benefit from 3D content creation, such as gaming, animation, and virtual/mixed reality. It enables users to create high-quality and diverse textures for 3D objects using natural language, which is more intuitive and flexible than manual editing. Additionally, it facilitates the creative process of 3D artists by allowing quick iterations and exploration of different styles and variations.

Conclusion

In summary, Meta 3D TextureGen proved effective for generating high-quality textures for 3D objects based on textual descriptions. The researchers claimed that their method brings texture generation closer to an applicable tool for 3D artists and general users to create diverse textures for assets in gaming, animation, and virtual reality (VR). Moving forward, they suggested generating material maps, such as tangent normal, metallic, and roughness, and reducing dependence on 3D datasets to improve their technique.