Exploring Pareidolia: AI Models Bridge the Gap Between Human and Machine Face Detection

By Dr Silpaja Chandrasekar, PhDReviewed by Joel ScanlonOct 3 2024

New research uncovers how AI systems can better detect face pareidolia, inspired by humans' evolutionary ability to recognize faces—both human and animal.

Research: Seeing Faces in Things: A Model and Dataset for Pareidolia. Image Credit: Laura Polo / Shutterstock

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

In an article recently submitted to the arXiv preprint* server, researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) explored face pareidolia from a computer vision perspective, introducing a dataset called "Faces in Things," which included 5,000 images with human-annotated pareidolic faces. They found a significant behavioral gap between human and machine face detection capabilities, potentially linked to the hypothesized evolutionary necessity for humans to recognize not only human but also animal faces. The study proposed two statistical models—a Gaussian model and a feature-based model—of pareidolia, confirming predictions about the image conditions that induce this phenomenon.

Background

Past work highlighted the evolution of face detection from the Viola-Jones detector to modern convolutional neural networks (CNNs), achieving near-human performance. Research indicated that face pareidolia is processed in the brain's fusiform face area, with evidence showing that even minimal facial cues trigger significant cognitive responses. Despite extensive studies in face detection, the specific exploration of face pareidolia has been limited, lacking a large-scale dataset dedicated to this phenomenon.

Pareidolic Dataset Creation

To address the gap in face pareidolia research, the team sampled candidate pareidolic images from the large-scale artificial intelligence open network 5 billion (LAION-5B) dataset containing 5.85 billion contrastive language-image pretraining (CLIP)-filtered image-text pairs. They constructed a raw image set using CLIP retrieval with queries like "pareidolia" and "Faces in Things."

After downloading and deduplicating images, they downsampled them to 512 × 512 pixels with white-space padding to preserve the aspect ratio. The visual geometry group (VGG) image annotation tool manually annotated the images, excluding those featuring actual human or animal faces. The annotations included bounding boxes of pareidolic faces and basic facial attributes. The final dataset was randomly divided into training (70%) and testing (30%) sets, referred to as the "Pareidolic" dataset.

“Face pareidolia has long fascinated psychologists, but it’s been largely unexplored in the computer vision community,” says Mark Hamilton, MIT PhD student in electrical engineering and computer science, CSAIL affiliate, and lead researcher on the work. “We wanted to create a resource that could help us understand how both humans and AI systems process these illusory faces.”

Exploring Face Pareidolia Datasets

The analysts utilized several additional datasets to explore face pareidolia, including the Pareidolic dataset, WIDER FACE, and AnimalWeb. The WIDER FACE dataset is a prominent face detection benchmark, featuring 32,203 images with a vast array of facial expressions, poses, and lighting conditions organized into 61 event classes.

The AnimalWeb dataset contains 22,451 annotated faces from 334 species, allowing for robust analysis of animal facial features. The researchers converted key facial landmarks into bounding boxes to standardize the dataset, which they split into training and testing sets for further analysis.

To investigate the effects of data augmentations on pareidolia, the team corrupted images from the WIDER FACE dataset using various methods, including Sobel filtering, which is known to lessen reliance on texture information. The RetinaFace model, renowned for its accuracy in detecting faces, was fine-tuned on different datasets, with attention to how training data influenced the model's ability to detect pareidolic faces. In particular, the inclusion of animal faces was found to significantly improve detection rates, highlighting the evolutionary hypothesis that recognizing diverse facial structures, including non-human faces, might provide adaptive advantages for detecting threats and opportunities in the environment.

Through their experiments, the analysts observed that fine-tuning the RetinaFace model on animal faces substantially enhanced its pareidolic face detection capabilities. They reported that adding pareidolic and animal images in training data improved performance compared to using either dataset alone.

A key finding from feature analysis showed that animal and pareidolic faces clustered closely together in feature space, suggesting a deeper connection between these two categories, compared to human faces. This clustering supports the hypothesis that training models on a broader range of facial types, particularly non-human faces, could improve their ability to recognize pareidolia.

Pareidolia Detection Models

The study on pareidolia introduces two formal models to describe this phenomenon: a Gaussian model and a feature-based model, both of which describe how image complexity influences pareidolia detection. The Gaussian model simulates image generation as a sum of independent normal modes and predicts that pareidolic faces are most likely to emerge when image complexity hits an optimal balance—a “Goldilocks” zone where there are enough features to suggest a face, but not so many that they create noise. The second model incorporates higher-level features and posits that detecting a pareidolic instance requires identifying specific features in designated spatial regions. This model also predicts a similar "peak pareidolia" effect, where detection probability is highest in medium-complexity images.

The investigators validated their models through experiments with human subjects and machine learning (ML) models, revealing a distinct peak in pareidolia detection at a specific filter width in frequency space. Both models confirmed that human and machine detectors are most likely to identify pareidolic faces in images of intermediate complexity, reinforcing the idea of an optimal range of visual complexity for pareidolia.

Conclusion

To sum up, the team made initial progress in mathematically modeling pareidolia and created a richly annotated dataset for face pareidolia. They demonstrated through experiments on modern face detectors that the detection of animal faces might contribute to the emergence of pareidolia in complex vision systems. The "Faces in Things" dataset was intended to help the community explore questions regarding pareidolic behavior, a key feature of human recognition systems. The researchers hope their findings and dataset will inspire further investigation into pareidolia and its potential applications for enhancing computer vision systems.

“This is a delightful paper! It is fun to read and it makes me think. Hamilton et al. propose a tantalizing question: Why do we see faces in things?” says Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering at Caltech, who was not involved in the work. “As they point out, learning from examples, including animal faces, goes only half-way to explaining the phenomenon. I bet that thinking about this question will teach us something important about how our visual system generalizes beyond the training it receives through life.”

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.