Vista3D: Unveiling the Hidden Dimensions of a Single Image

By Soham NandiReviewed by Joel ScanlonSep 25 2024

Unlock the unseen side of 3D modeling—Vista3D generates stunning, detailed 3D objects from single images in just minutes, pushing the boundaries of gaming, virtual reality, and more.

Research: Vista3D: Unravel the 3D Darkside of a Single Image. Image Credit: Master1305 / Shutterstock

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

In an article recently submitted to the arXiv preprint* server, researchers introduced Vista three-dimensional (Vista3D), a framework for swift, consistent 3D object generation from single images. Leveraging a dual-phase process, they used a two-phase approach, generating initial geometry with Gaussian splatting and refining it with a signed distance function (SDF). By combining two-dimensional (2D) and 3D diffusion priors, Vista3D effectively captured objects' visible and hidden aspects, achieving high-quality, diverse 3D models in as little as five minutes.

Background

The study of generating 3D models from 2D images has gained prominence due to advancements in 3D generative models, which offer applications in areas such as gaming, virtual reality, and more. Previous methods, such as sparse-view reconstruction and large-scale 2D diffusion models, struggled with issues like blurred 3D outputs and limited texture diversity. These limitations arose from insufficient 3D data and the neglect of unseen object aspects.

Vista3D addressed these gaps by introducing a dual-phase strategy combining Gaussian splatting and SDFs to efficiently generate 3D objects with diverse and consistent textures from a single image. Vista3D also introduced a novel angular-based texture composition approach, ensuring improved structural integrity and texture accuracy while capturing both visible and obscured object dimensions.

This framework significantly improved 3D generation quality by blending 2D and 3D diffusion priors for high-fidelity, rapid results. Through this method, Vista3D filled the existing gaps in previous approaches by offering a unified, efficient solution for consistent and detailed 3D model generation.

3D Object Generation with 2D Diffusion Priors

Using diffusion priors, the methodology outlined a novel framework for generating detailed 3D objects from single 2D images. The process began with generating coarse geometry using 3D Gaussian splatting, which created a basic 3D structure quickly but required significant optimization to densify and refine it. In this initial stage, a Top-K gradient-based densification method was introduced to stabilize the optimization process, along with regularization techniques to control the geometry’s scale and transmittance.

The next stage involved refining the coarse geometry into an SDF using a differentiable hybrid mesh representation to smooth out surface artifacts. This refinement utilized FlexiCubes, a cutting-edge differentiable isosurface representation, to make local adjustments to the geometry. The texture was learned using a disentangled texture representation, which separated texture supervision for improved performance in unseen views.

To enhance the diversity of unseen views, the framework incorporated two advanced diffusion priors, one from Zero-1-to-3 XL and another from Stable Diffusion. A gradient constraint method was applied to balance the contributions of both priors, ensuring consistency in the 3D model while introducing diversity in the unseen aspects of the object. This method efficiently generated high-fidelity 3D objects, addressing limitations in conventional rendering techniques.

Experimental Setup and Results

The Vista3D framework was designed for rapid and efficient 3D object generation from 2D images using a coarse-to-fine optimization approach. Initially, a coarse geometry was learned by preprocessing images with a Segment Anything Model (SAM), where 3D Gaussians were optimized over 500 steps, gradually refining the object’s geometry and texture.

Pruning and densification techniques ensured that transparent Gaussians remained unaffected while regularization enhanced geometry and texture consistency. During mesh refinement, FlexiCubes with a grid size of 80³ were used to fine-tune the geometry, and the texture was enhanced using hash encodings and a multilayer perceptron (MLP) model.

Vista3D-S completed this process within five minutes, while Vista3D-L took up to 20 minutes, incorporating additional diffusion priors for more detailed textures. The framework introduced angular diffusion prior composition for handling unseen object views, which further enhanced both geometry and texture consistency.

Comparative studies showed that Vista3D surpassed other methods like Magic123 and DreamGaussian in generating superior textures and geometries. Quantitative experiments using CLIP-similarity and other metrics on datasets like RealFusion and Google Scanned Object (GSO) further highlighted its superior performance. In addition, user studies and ablation experiments confirmed that the coarse-to-fine pipeline and disentangled texture learning were essential for achieving state-of-the-art 3D object generation with minimal artifacts.

Conclusion

In conclusion, the researchers introduced Vista3D, a framework for efficiently generating 3D objects from single 2D images using a dual-phase approach. First, coarse geometry was created through Gaussian splatting and then refined with an SDF. This method skillfully blended 2D and 3D diffusion priors to ensure fast, high-quality, detailed 3D models, capturing visible and hidden object aspects.

By incorporating angular-based texture composition, the framework achieved high-fidelity 3D object generation within minutes, addressing gaps in previous methods that struggled with texture diversity and unseen object details. Vista3D significantly improved on earlier techniques, offering a unified solution for swift, high-quality 3D model generation, validated through extensive user studies and comparative performance evaluations with methods like Magic123 and DreamGaussian.

The framework's innovative use of diffusion priors, advanced disentangled texture learning, and FlexiCubes for surface refinement ensured superior results, opening new possibilities for various industries such as virtual reality, gaming, and more.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.