Enhancing Road Safety Using a CNN-LSTM Model for Driver Sleepiness Detection

By Soham NandiReviewed by Susha Cheriyedath, M.Sc.Jan 3 2024

In an article published in the journal Information, researchers focused on enhancing road safety by utilizing a deep learning (DL) model, specifically a convolutional neural network – long short-term memory (CNN-LSTM) network, for driver sleepiness detection. The study employed various algorithms, such as EM-CNN, VCG-16, GogLeNet, AlexNet, and ResNet50, and it demonstrated that CNN-LSTM performed with superior accuracy. The model analyzed video clips to identify signs of drowsiness, aiming to proactively notify drivers and prevent accidents caused by fatigue-related issues.

Background

The advent of advanced technologies has ushered in a new era in automotive safety, particularly at the intersection of the Internet of Things (IoT) and driver assistance systems. Driver drowsiness or driver fatigue poses a serious threat to road safety and is a leading cause of accidents and injuries. As driving conditions vary with factors such as time of day, driver profiles, and travel purposes, identifying effective strategies for mitigating the risks associated with drowsy driving becomes imperative.

Previous research emphasized the requirement for proactive measures to address this public health concern. Traditional approaches implemented to tackle the problem of driver drowsiness include self-reporting through instruments like the Driver Behavior Questionnaire and psychological monitoring through devices attached to the driver's skin. However, the limitations of these methods led to the exploration of innovative solutions with the help of IoT and deep learning technologies. Previous studies in drowsiness detection have employed various methodologies, including physiological measurements, vehicle-based analyses, and behavioral monitoring.

Physiological methods have shown promise but come with the inconvenience of attaching devices to the driver's skin. Vehicle-based methods analyzing steering wheel movements or eye states face challenges related to reliability and may result in false positives. Behavioral recognition using CNNs has been explored, but existing models often lack focus on specific driver actions, particularly those associated with drowsiness.

This paper introduced a novel approach that combined U-Net-based segmentation, CNN-LSTM modeling, and IoT principles. Through integrating facial movement analysis and DL, researchers contributed to the ongoing efforts to make driving safer by effectively identifying and addressing drowsy driving risks.

Proposed Method ⁠⁠⁠⁠⁠⁠⁠

The proposed driver drowsiness detection system utilized OpenCV and Dlib libraries to assess eye closure, yawning, and facial expressions, crucial indicators of drowsiness. Employing a U-Net-based segmentation approach, the model processed frames from a video clip, extracting the driver's body region. The system incorporated a multitasking framework for facial feature detection and employed cascaded convolutional networks for precise alignment. The architecture integrated convolutional and LSTM layers, enhancing the accuracy of drowsiness classification. Regular testing on diverse datasets ensured the system's reliability in varied conditions, contributing to an effective real-world drowsiness detection solution.

The study utilized a dataset from Kaggle containing frontal images of drivers captured over 55 seconds at 12 frames per second. The dataset was normalized, and experiments were conducted across four scenarios related to open eyes, closed eyes, and open mouths. Feature extraction involved using a CNN-LSTM model with specific configurations and detecting drowsiness relied on monitoring PERCLOS (Percentage of Eye Closure over Time) and POM (Probabilistic Output Model).

The PERCLOS was calculated based on closed-eye frames over total frames, and a threshold of 0.25 indicates drowsiness. The proposed model combined the strengths of CNNs for spatial features and LSTMs for temporal patterns in analyzing real-time video feeds of drivers' faces. The architecture demonstrated efficacy in drowsiness detection, with potential applications beyond, such as integration into smart vehicles. The algorithm involved preprocessing images, combining them with model inputs, retrieving results from CNN layers, flattening dimensions, applying CNN-LSTM layers, and using specific activation functions.

Experimental Results

The proposed approach achieved an accuracy greater than 98%, leveraging real-world images of drivers in various conditions. The dataset included 640 images for training and 475 for testing the closed mouth state. The implementation process involved training the model with 100 epochs on a system with six-core processors. CNN-LSTM outperformed various models, including EM-CNN, VGG-16, GoogLeNet, AlexNet, and ResNet50, achieving a training accuracy of 98.70% and testing accuracy of 98.80%. Evaluation metrics included accuracy, loss, precision, recall, and F1 score, with CNN-LSTM displaying superior results. The model's sensitivity to mouth state compared to eye state was highlighted, emphasizing the mouth's precision in indicating drowsiness. The evaluation considered different scenarios, including "No Glasses," which yielded the highest accuracy, and limitations were identified, such as challenges with binarization for individuals with dark skin.

The authors discussed the importance of loss function optimization in training robust drowsiness detection models and emphasized the significance of data acquisition, capturing diverse facial movements for accurate detection. The researchers also discussed limitations related to binarization challenges for dark skin tones and the impact of background homogeneity on system reliability. Recommendations for anomaly detection techniques, hyperparameter optimization, and handling challenging scenarios like glare were provided. It contributed to the understanding of loss function optimization and the challenges associated with diverse datasets and real-world conditions in drowsiness detection systems.

Conclusion

In conclusion, integrating IoT technology with facial movement analysis has advanced road safety, particularly in autonomous driver sleepiness detection. This study introduced a CNN-LSTM model for real-time monitoring of driver drowsiness, outperforming other techniques. Future work aims to enhance portability, mitigate accident risks, and potentially incorporate attention modules to improve feature interaction and nuanced categorization, boosting the model's overall performance in drowsiness detection.