Advancing Deep Learning with Differentiable Visual Computing

By Samudrapom DamReviewed by Susha Cheriyedath, M.Sc.Nov 22 2023

In an article recently published in the journal Nature Machine Intelligence, researchers reviewed the advances and challenges of the constituent parts of a complete differentiable visual computing (DVC) pipeline.

Background

Visual computing methods designed for computer graphics applications primarily synthesize information about virtual and physical worlds using algorithms optimized for spatial computing. Visual computing is utilized to render the world through optical techniques, analyze geometry, and physically simulate fluids and solids

Deep learning allows the development of general algorithmic models, which eliminates the need for a first-principles-based approach to solve problems. Deep learning is based on highly parameterized neural network architectures and gradient-based search algorithms, which can search the large parameter space efficiently for optimal models.

Although visual computing provides a robust inductive bias about real-world phenomena dynamics, it cannot adapt its mathematical models based on real-world phenomena observations, which prevents the direct integration of visual computing into larger deep learning-based systems.

Similarly, deep learning requires diverse and high-quality data observations of the underlying dynamics, an adequately expressive neural network, and time for training the model to formulate relationships between dynamics and observations. DVC can be used to address this issue as it combines explicit, compact, and accurate models of reality with system parameterization and gradients required to adapt the model parameters/using gradient-based optimization. Thus, the combination of DVC- and deep learning-based methods can significantly increase the data efficiency, accuracy, and speed with which machine learning (ML) can be used in inference problems in real-world physical systems.

The study

In this paper, the authors proposed a holistic and unified DVC pipeline that combines advancements in differentiable geometry, physics, and animation for visual content generation and efficiently understanding and improving the visual content using gradient-based information. Specifically, they reviewed the advances and challenges of three key aspects of computer graphics, including rendering, animation, and geometry.

The classic computer graphics pipeline progresses by instantiating geometry, animating the geometry as desired, and finally rendering it for visual appeal. DVC allows every computer graphics process and the end-to-end pipeline to be differentiable.

Geometry: The pipeline starts with geometry, and the definition of geometric objects is always mathematical with algorithmic or direct means of specification. Recently, CAD kernels were introduced that allow geometry specification process differentiation and manufactured performance optimization at design time.

Geometric representations fall along three axes: discrete versus continuous, volumetric versus surface, and explicit versus implicit. Geometric representations are also selected based on the gradient information type required for the downstream task.

Generating continuous derivatives from discrete representations is a major issue, as most geometric representations are discrete. Learning continuous representations, output smoothing, and domain smoothing are the three core methods for reasoning about discrete geometry smoothly.

Domain smoothing involves the transformation of a discrete variable into a continuous variable, while in output smoothing, consuming algorithms primarily smooth the domain to interpolate the space between discrete regions. In differentiable geometry processing, neural operators can replace entire or parts of algorithms with domain-agnostic differentiable models that can be embedded in a DVC algorithm or learned offline.

These operators can provide useful gradients and still reason about discrete structures. The common approach is embedding discrete inputs as continuous tensors that encode the geometry structure. However, deriving relevant gradients in geometry processing has several challenges, including combining the learned and analytical models most effectively and continuously reasoning about discontinuous structure.

Animation: Physically based animation/physics simulation is used for computationally visualizing, analyzing, and predicting the interaction and motion of objects in the world. The authors primarily focused on time-varying, dynamic simulators in this review.

The emerging differentiable physics simulators can provide a means of both predicting reality and understanding that reality better. Learning-based and explicit methods are the two common strategies for constructing differentiable physical simulators.

Explicit methods derive simulators and their gradients from first principles by programming a model of a physical system time evolution and then deriving its gradients algorithmically. Learning-based methods focus on learning a state-transition model from the observed real-world or simulated data.

Although less domain knowledge is required in these methods, they often depend on physics-inspired neural network architectures catered to the matter arrangement or state evolution structure. In both cases, a complete dynamic simulation is developed by applying the state transition model recursively for the desired simulation duration. However, most simulation domains involve hard boundary conditions/hard constraints, which can be repeated several times throughout a simulation. Thus, selecting a relaxed, smoothed contact model is crucial to alleviate gradient non-smoothness.

Rendering: Differentiable rendering approaches can solve different sub-problems in inverse inference, such as numerical estimation of a synthesized image gradient corresponding with the geometrical changes, such as viewpoint changes owing to the several possible placements of a camera sensor, lighting changes due to multiple scattering of light energy in an environment, and occlusion.

These events are continuously differentiable and smooth excluding sparse event points. In differentiable rendering, open research challenges can be categorized along algorithmic axes and numerical methods.