Mindful Machines: Deep Neural Networks Master the Art of Predicting Face Memorability

By Muhammad OsamaReviewed by Susha Cheriyedath, M.Sc.Jan 22 2024

In an article published in the journal Scientific Reports, researchers from the Western University and the Vector Institute for Artificial Intelligence, Canada, developed an innovative method for predicting the memorability of facial photographs using deep neural networks. They used a large database of natural face photographs and their memorability scores to train and test different models.

The research highlighted that the proposed technique outperformed previous methods and achieved near-human consistency in predicting face memorability. Moreover, it demonstrated that the new methodology could handle both oval-shaped and square-shaped face images.

Background

A deep neural network is a type of artificial neural network (ANN) that has more than two layers. It is a powerful model capable of learning complex patterns from large amounts of data and making predictions. These networks are used to solve complex problems such as image processing, computer vision, natural language processing, and transfer learning. They are inspired by biological neural networks and consist of a collection of layers to perform a specific task. Each layer has a collection of nodes that operate together. The additional layers help the model understand problems better and provide optimal solutions to complex projects.

Face photographs are ubiquitous in daily life, especially on social media platforms. People encounter many faces every day, but only some of them get registered in their minds, while others are easily forgotten. Image memorability is the possibility or probability that a viewer or observer will remember an image after its only one exposure.

Previous studies have shown that image memorability is an intrinsic property of an image, meaning that some images are inherently more or less memorable than others, regardless of the preference or bias of observers. Furthermore, image memorability is highly consistent across individuals and over time, suggesting the existence of common features that make an image memorable.

Past research has attempted to predict image memorability using various features, such as global descriptors, scene categories, or attention maps. However, these techniques were not fully automatic and needed manual tuning. Recently, researchers employed deep neural networks to predict image memorability. These models demonstrated excellent performance on scene and object images but have struggled to predict the memorability of face photographs, a special category of images with high social and emotional relevance.

About the Research

In the present paper, the authors designed a new methodology to predict the memorability of face photographs employing deep neural networks. They used a 10k US adult face database containing 10,168 natural face photographs and their memorability scores obtained from a large-scale online experiment. The memorability scores reflect how likely an observer is to detect the repetition of a face photograph after seeing it once among a stream of other face photographs.

The researchers assessed three state-of-the-art memorability models that were trained on a general dataset of scene and object images called the large-scale memorable moments dataset (LaMem). They found that these models performed poorly on the face photographs, indicating that face memorability differs from scene and object memorability. They then proposed seven different models to estimate the memorability of face photographs based on different architectures and pre-training strategies. Moreover, they fine-tuned these models on the face dataset and compared their performance with the previous methods and with human consistency.

Research Findings

The outcomes showed that the introduced models outperformed the previous methods and achieved near-human consistency in forecasting face memorability. The authors emphasized that the models pre-trained on a face recognition task and fine-tuned on a face memorability task performed better than the models pre-trained on a general image classification task and fine-tuned on a face memorability task. This suggests that the models pre-trained on face images are more effective in extracting face features relevant for memorability. They also showed that their models could predict the memorability of both square-shaped and oval-shaped face images, demonstrating their robustness and generalizability.

The applications of predicting the memorability of face photographs are vast and varied, spanning different fields such as social media, advertising, education, security, and entertainment. For instance, this method can be used by social media users to select the most memorable profile pictures or posts, by advertisers to design more effective campaigns or logos, by educators to create more memorable learning materials, by security agencies to identify the most memorable suspects or witnesses, and by game developers to create more memorable characters or scenarios.

Conclusion

In summary, a deep neural networks-based novel approach effectively and efficiently predicted the memorability of faces in photographs. It surpassed previous methods and achieved near-human consistency in estimating face memorability. Moreover, it demonstrated the capability to handle face images of different shapes and sizes, presenting potential applications in various fields related to face photographs and human memory.

The authors acknowledged challenges and limitations in their development. They suggested that future work should include other types of images, such as caricatures, sketches, or paintings. Additionally, they proposed investigating the factors influencing face memorability, including emotions, expressions, or identities.