Meta's 3DGen Revolutionizes Text-to-3D Generation

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Jul 16 2024

In an article recently posted to the Meta Research website, researchers introduced Meta 3D generation (3DGen), a state-of-the-art pipeline for text-to-3D asset generation. 3DGen created 3D assets with high prompt fidelity and high-quality shapes and textures in under a minute. It supported physically based rendering (PBR) for real-world applications and allowed generative retexturing of existing 3D shapes with additional textual inputs.

Combining Meta's 3D asset generation (Meta 3D AssetGen) and Meta's 3D texture generation (Meta 3D TextureGen), 3DGen represented 3D objects in view, volumetric, and ultraviolet (UV) space, achieving a win rate of higher accuracy over single-stage models. Researchers showed that 3DGen outperformed industry baselines in prompt fidelity and visual quality for complex prompts while being significantly faster.

Background

There is ample literature on text-to-3D and text-to-texture generation. Key methods train 3D generators on limited 3D datasets or use image/video-based generators trained on billions of samples. Some approaches focus on multi-view consistency and 3D reconstruction using neural radiance fields (NeRF)) and similar techniques.

Additionally, there is significant work on texture generation, utilizing contrastive language–image pre-training (CLIP) guidance, generative adversarial network (GAN)- like approaches, and diffusion in UV space.

3DGen Method Overview

The method begins with an overview of 3DGen's core components: AssetGen (stage I) and TextureGen (stage II). TextureGen serves as the text-to-texture generator within stage II, focusing on producing textures that align with textual prompts for given 3D shapes. This process involves generating multiple views of the object, projecting these views onto textures, and integrating them to create a final, detailed texture output.

At stage I, AssetGen acts as the text-to-3D object generator, creating the 3D mesh and corresponding texture based on textual input. This stage employs a sequential approach, generating initial object views and reconstructing the 3D mesh and texture. Each stage leverages specific neural networks trained to handle these tasks efficiently and effectively.

Integrating these stages under Meta 3D Gen aims to enhance the quality and fidelity of text-to-3D generation. By combining the specialized capabilities of TextureGen in refining textures and AssetGen in generating accurate 3D meshes, the approach ensures robust performance in creating complex 3D assets directly from textual descriptions.

Meta 3D Gen Evaluation

In the experimental evaluation, Meta 3D Gen is compared against several prominent industry solutions for text-to-3D asset generation. The review assesses the quality and fidelity of generated outputs through user studies across diverse categories. Meta 3D Gen is benchmarked against platforms like common sense machines cube 2.0 (CSM Cube 2.0), tripo3D by Tripo artificial intelligence (Tripo3D), Rodin gen-1 by Deemos (Rodin Gen-1), meshy version 3 (Meshy v3), and a third-party T23D generator. Each uses distinct methodologies and APIs tailored for generating 3D assets from textual descriptions, reflecting varying capabilities and runtime efficiencies.

User studies focus on prompt fidelity and visual quality, with annotators comprising general users, professional 3D artists, and developers. The studies utilize a set of 404 text prompts covering objects, characters, and complex compositions originally curated by dreamfusion. Results highlighted Meta 3D Gen's superiority in fidelity across both stages of generation, particularly in comparison to the strongest competitor, the third-party T23D generator.

Qualitative insights further underscore findings, showing Meta 3D Gen excels as scene complexity increases, particularly in scenarios involving intricate compositions of characters and objects. Visual comparisons demonstrate Meta 3D Gen's advancements over industry baselines in texture quality, geometric fidelity, and handling complex prompts. These comparisons highlight the robustness of the approach, particularly in scenarios where competitors struggle with geometry artifacts, texture seams, or incomplete representations.

In conclusion, Meta 3D Gen is a formidable solution for text-to-3D generation, excelling in quantitative benchmarks and qualitative assessments across a spectrum of complexity levels. Its performance underscores potential utility across diverse applications, from gaming and animation to virtual prototyping and creative design.

Conclusion

To summarize, 3DGen introduced a unified pipeline integrating Meta's foundational generative models for text-to-3D generation with texture editing and material generation capabilities, AssetGen and TextureGen, respectively. Combining their strengths, 3DGen achieved high-quality 3D object synthesis from textual prompts in less than a minute.

When assessed by professional 3D artists, the output of 3DGen was preferred most of the time compared to industry alternatives, particularly for complex prompts, while demonstrating speeds from 3× to 60× faster. The integration of AssetGen and TextureGen set a promising research direction, focusing on generation in view and UV space and end-to-end iteration over texture and shape generation.

This streamlined approach enhances efficiency in 3D asset creation and highlights potential advancements in real-time applications such as gaming, virtual prototyping, and creative design. By refining the synthesis process and optimizing resource utilization, 3DGen paves the way for future innovations in AI-driven content generation, catering to diverse industry needs with enhanced speed and fidelity.