CV Strategies in Deep Learning for Psychiatric Disorder Diagnosis

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Oct 5 2023

In a paper published in the journal Scientific Reports, researchers explored the impact of cross-validation methods on the diagnostic performance of deep-learning-based computer-aided diagnosis (CAD) systems using augmented neuroimaging data. They utilized electroencephalogram (EEG) data from post-traumatic stress disorder patients and controls, expanding it with different window sizes.

Four cross-validation (CV) approaches were employed, and two convolutional neural network-based models were used to assess diagnostic performance. The study revealed that data augmentation improved performance, but incorrect CV methods led to inflated results due to data leakage. Researchers emphasized the importance of employing the correct CV strategies (subject-wise CV (sCV) and overlapped Subject-wise CV (oSCV)) for developing robust CAD systems for psychiatric disorders using deep learning and data augmentation.

Background

Developing CAD systems using neurophysiological features has gained prominence in psychiatry. These systems aim to enhance the accuracy of diagnosing psychiatric patients by reducing potential errors associated with traditional diagnosis methods, which rely on clinical expert interviews. Among various neuroimaging modalities, EEG is a valuable tool for CAD systems. EEG-based neurophysiological features effectively capture abnormal functional traits in psychiatric patients, resulting in superior diagnostic performance when distinguishing them from healthy controls (HCs). Recent efforts have turned to state-of-the-art deep-learning algorithms to bolster the reliability and diagnostic accuracy of EEG-based CAD systems for psychiatric patients.

Related Work

Previous research in CAD for psychiatric patients has turned to deep learning and EEG data to improve diagnostic accuracy. Data augmentation, especially the cropping method, has played a pivotal role in increasing the available data for training deep-learning models. Researchers have enhanced diagnostic performance by segmenting EEG time-series data into non-overlapping segments. However, cropping-based data augmentation requires careful consideration to avoid data leakage, which can inflate diagnostic accuracy. Some recent CAD studies need to look into separating training and test data, leading to data leakage.

Proposed Method

Participants: Researchers recruited 77 PTSD patients and 58 HCs from Inje University Paik Hospital's Psychiatric Department. Clinical experts assessed three psychiatric symptoms: The impact of Event Scale-Revised (IES-R) for traumatic event stress, the Beck Depression Inventory (BDI) for depression, and the Beck Anxiety Inventory (BAI) for anxiety. They recruited HCs without psychiatric medical history from the local community. Participants' demographic data and symptom scores were collected and approved by the Institutional Review Board of Inje University Ilsan Paik Hospital.

EEG Recording and Preprocessing: Resting-state EEG data were recorded with 64 Ag/AgCl electrodes using a NeuroScan SynAmps2 system. Researchers removed eye-related artifacts using regression and visually inspected and removed gross artifacts. They downsampled the EEG data to 200 Hz from 1000 Hz and used 60 seconds of EEG data for analysis to enhance computational efficiency. Data augmentation was performed by cropping EEG data into segments of different window lengths, with each element treated as an independent trial.

CV Strategies: The validation used four CV strategies: sCV, osCV, tCV, and overlapped tCV (otCV). In sCV, they utilized all augmented trials from a single participant for training and test data. On the other hand, in tCV, they randomly divided the extended problems of a participant into training and test data. This strategy used a voting strategy with a 60% threshold in sCV, and they applied the same approach to trials with a 75% overlap, creating osCV and similarly implementing tCV and otCV. They performed a 10×10-fold CV to estimate diagnostic performance for these strategies.

Convolutional Neural Network (CNN) Architectures: Two CNN models, CNN-13 and EEGNet, were used to evaluate diagnostic performance. CNN-13 comprised 5 convolutional layers, five pooling layers, and three fully-connected layers, while EEGNet included three convolutional layers, two pooling layers, and one fully-connected layer. Both models used specific activation functions, learning rates, and dropout rates and calculated diagnostic performance using balanced classification accuracy.

Feature Distribution, Statistical Analysis: This approach analyzed feature distribution to understand the data leakage problem by extracting 992-dimensional features and then reducing them to two dimensions using t-stochastic neighbor embedding (t-SNE). This method used Linear discriminant analysis (LDA) to compute decision boundaries. Statistical analysis included Friedman's test to evaluate differences among CV methods for each window length and the Wilcoxon rank sum test for pairwise comparisons, with Bonferroni correction for adjusted p-values.

Experimental Results

The study found that data augmentation improved the diagnostic performance of CAD systems for psychiatric disorders. Classification accuracy increased as window lengths decreased, with augmented data consistently outperforming the original data. For the CNN-13 model, osCV achieved the highest accuracy, surpassing 80%. EEGNet performed well with tCV and otCV. Feature distribution analysis showed that tCV and otCV provided better separation between patient and control group features, highlighting the importance of choosing an appropriate CV strategy to avoid data leakage and optimize classification performance.

 Conclusion

To summarize, data augmentation has become increasingly important in enhancing the performance of deep-learning algorithms, mainly when training data is limited. This study investigated the impact of different CV strategies on the performance of EEG-based CAD systems within the context of data augmentation. The results demonstrate that data augmentation can improve CAD system performance.

Researchers must still exercise caution in choosing the correct CV method to prevent data leakage and avoid inflating classification accuracy. sCV with a voting strategy emerged as a reliable approach to obtain accurate classification performance after data augmentation. This study offers valuable guidance for researchers working on neuroimaging-based CAD systems for psychiatric disorders who may be new to data augmentation and deep-learning methods.