UIBVFEDPlus-Light: Virtual facial expression dataset with realistic lighting

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Oct 4 2023

In a paper published in the journal PLOS ONE, researchers emphasized the critical impact of lighting conditions on automatic human expression recognition by introducing the University of the Balearic Islands Virtual Facial Expression Database (UIBVFED) Plus-Light, an extension of the UIBVFED virtual facial expression dataset consisting of 100 virtual characters portrayed under four lighting setups. User-friendly software, accompanied the database, facilitated interactive exploration of expressions and lighting adjustments, providing new avenues for studying emotion recognition techniques under realistic lighting scenarios.

Background

Facial Emotion Recognition (FER) represents a persistent challenge in computer vision, gaining relevance in research and various visual art forms such as Computer Graphics (CG) art. Despite the prevalence of FER datasets, researchers have often overlooked lighting conditions as a critical factor. Existing facial emotion databases typically manage character lighting, aiming for lighting-independent datasets. Nevertheless, studies have demonstrated that heightened illumination enhances three-dimensional (3D) facial expression recognition. Consequently, offering recommendations for optimal lighting levels can contribute to lighting standard development.

Previous studies have categorized facial expression databases into the controlled laboratory and uncontrolled environment groups. The former includes databases like Karolinska Directed Emotional Faces (KDEF) and Extended Cohn-Kanade (CK+). At the same time, the latter encompasses datasets like Static Facial Expression in the Wild (SFEW) and AffectNet, offering details on composition, camera position, origin, annotation types, and affect modeling. Among controlled databases, the Multiple Posing and Illumination Database (Multi-PIE) uniquely considers lighting, featuring 18 lighting conditions and 15 cameras for diverse viewpoints. Past research has also explored lighting's impact on emotion perception, examining techniques like the three-point lighting system and different lighting directions for creating visual effects.

Proposed Method

UIBVFED, a virtual facial expression database, pioneers categorizing up to 32 facial expressions and contains 640 facial images featuring these expressions portrayed by 20 virtual characters. The dataset encompasses a diverse demographic, including 10 men and 10 women of various ages and ethnicities. These expressions align with Faigin's classification of the six universal emotions. The database offers detailed texture resolution and utilizes the Unity 3D development environment.

It maintains a neutral scene with three-point classical lighting and introduces a precise Facial Action Coding System (FACS) alignment for objectively labeling images. The newly introduced UIBVFEDPlus-Light database extends the original UIBVFED. It now comprises 100 virtual characters, diversifying the sample compared to its predecessor.

The characters, created using Autodesk Character Generator, can each reproduce the 32 facial expressions categorized by Faigin.  Additionally, the database introduces lighting variations using a simplified eight-light system, resulting in four distinct lighting configurations. These configurations involve key, fill, and backlights, affecting the character's shadows and appearance. The lighting options aim to enhance the realism and versatility of the dataset.

The UIBVFEDPlus-Light software application, an extension of the original, allows interactive exploration of expressions and facilitates control over lighting options, enabling users to adjust expression intensity and lighting configurations. Furthermore, the database now includes precise 3D spatial coordinates for 51 facial landmarks, enhancing its utility for expression recognition research. Overall, UIBVFEDPlus-Light enhances the original database by expanding the character set, introducing lighting variations, and providing additional tools for researchers and developers in facial expression analysis and emotion recognition.

Study Results

An experimental evaluation using a Convolutional Neural Network (CNN) previously trained on the UIBVFED dataset, which featured 20 synthetic avatars capable of expressing the six universal emotions plus the neutral emotion, achieved an overall accuracy of 0.88. To evaluate the potential of the new UIBVFEDPlus-Light dataset, researchers trained and tested the same CNN model with facial expression images generated under the lighting configuration from above without contrast, resulting in an improved overall accuracy of 0.93, signifying enhanced recognition performance. The dataset's expansion from 660 to 3300 images likely played a significant role in this performance boost, showcasing the importance of lighting conditions in facial expression recognition.

Additionally, the CNN model trained on images with the lighting configuration from above without contrast was tested with various lighting configurations, revealing the impact of lighting conditions on emotion recognition. Notably, the classifier's ability to recognize emotions varied significantly based on lighting, underscoring the need for further research to investigate this influence comprehensively. The study also applied the Local Interpretable Model-agnostic Explanations (LIME) method to gain insights into the model's behavior under different lighting conditions, shedding light on the critical features considered for classification. Despite the dataset's contributions, it is essential to acknowledge its limitations, including decontextualized images, potential intensity variations within expressions, and the reliance on Faigin's morphological descriptions without human validation, providing directions for future research and dataset enhancements.

Conclusion

To sum up, this work presents a valuable resource with implications for multidisciplinary fields that can benefit from facial expression virtual prototyping. It introduces the UIBVFEDPlus-Light, an extension of the UIBVFED database, offering a comprehensive dataset of virtual characters featuring 33 distinct facial expressions under various lighting conditions. This expansion not only aids in mitigating overfitting in facial expression recognition but also brings new challenges for FER techniques in real-world lighting scenarios. Additionally, artists can leverage this tool for a nuanced understanding of how lighting influences facial expressions across different characters. While the database has limitations, such as uniform deformation values for terms, ongoing work seeks to enhance character realism and incorporate movement into the research.