MST-DeepLabv3+ for Semantic Segmentation Remote Sensing Images

By Soham NandiReviewed by Susha Cheriyedath, M.Sc.May 8 2024

In an article published in the journal Nature, researchers proposed MST-DeepLabv3+, a novel model for high-precision semantic segmentation of high-resolution remote sensing images. By reducing model parameters with MobileNetV2, integrating the squeeze-and-excitation networks (SENet) attention mechanism, and enhancing transfer learning, MST-DeepLabv3+ achieved improved segmentation accuracy.

Tested on various datasets, including the International Society for Photogrammetry and Remote Sensing dataset(ISPRS) and Gaofen image dataset (GID), the model demonstrated significant performance gains, offering better edge information recognition and reduced parameter size for efficient classification results.

Background

Remote sensing technology has revolutionized regional survey methods, offering swift data acquisition and vast information retrieval. It finds applications in soil research, geological engineering, and land resources management. Semantic segmentation, a crucial aspect of remote sensing image analysis, transforms complex images into understandable feature classifications, aiding practical applications.

While traditional machine learning methods like decision trees and support vector machines have been employed, they struggle with nuanced feature extraction and fluctuating target scales. The advent of convolutional neural networks (CNNs) has enhanced segmentation accuracy, yet classic models often suffer from redundant computations and inefficient memory usage.

Previous CNN-based approaches, including fully convolutional networks (FCNs) and DeepLab networks, have made strides in semantic segmentation. However, they still exhibit limitations in capturing intricate image contours and efficiently processing high-resolution images. This paper addressed these shortcomings by proposing MST-DeepLabv3+, a novel model tailored for high-precision semantic segmentation of remote sensing images.

By replacing the complex Xception backbone with MobileNetV2, integrating the SENet attention mechanism, and incorporating transfer learning, MST-DeepLabv3+ aimed to enhance segmentation accuracy while reducing model complexity and computational overhead. This paper filled the gaps in existing methodologies by offering a lightweight yet efficient solution for remote sensing image interpretation, promising improved accuracy, and faster processing times, thus advancing the field of remote sensing analysis and interpretation.

Advanced Methods and Diverse Datasets

The researchers utilized three distinct datasets to evaluate the proposed MST-DeepLabv3+ model for semantic segmentation of remote sensing images: the ISPRS dataset, the GID dataset, and the Taikang cultivated land dataset. The ISPRS dataset encompassed Vaihingen and Postdam subsets, offering a diverse array of urban scenes.

To mitigate the dataset's size limitations, augmentation techniques like flipping and rotating were applied, resulting in a larger dataset for training and testing. Similarly, the GID dataset, derived from Gaofen-2 satellite imagery, provided ample samples for land cover classification. Cropping and partitioning were employed to create training and testing subsets.

Furthermore, the Taikang cultivated land dataset was custom-created from high-resolution Gaofen-1 satellite images, focusing on agricultural land use classification. Through careful selection and preprocessing, a sizable dataset was constructed to validate the model's real-world applicability. The proposed MST-DeepLabv3+ model introduced several innovations to enhance semantic segmentation accuracy.

Firstly, it replaced the computationally intensive Xception backbone with the lightweight MobileNetV2 network, facilitating faster processing of high-resolution images without compromising accuracy. Secondly, the incorporation of the SENet attention mechanism improved feature channel weighting, enhancing segmentation precision. Lastly, transfer learning from the ImageNet dataset bolstered the model's feature extraction capabilities, refining segmentation accuracy.

Evaluation of model performance encompassed various metrics, including mean intersection over union (MIoU), overall accuracy (OA), precision, recall, and F1-Score, providing a comprehensive assessment of segmentation quality. Visual inspection of segmentation results supplemented quantitative analysis, ensuring a thorough evaluation of the model's efficacy.

By leveraging these datasets and methodologies, the researchers demonstrated the effectiveness of the MST-DeepLabv3+ model in the semantic segmentation of remote sensing images, underscoring its potential for real-world applications in land cover classification and agricultural monitoring.

Experimental Results and Analysis

The experimental setup included a CentOS7.9 operating system running on an AMD EPYC 7402 central processing unit (CPU) and 8*NVDIA®GeForce®RTX 3090 graphics processing units (GPU) with 8*24 gigabyte (GB) video memory, using pytorch3.6 framework. The model training utilized a batch size of eight and 100 iterations, reaching convergence at the maximum iteration limit. The Adam optimizer with a basic learning rate of 0.0005 was employed to dynamically adjust learning rates, enhancing model convergence.

Comparative analysis with DeepLabv3+, pyramid scheme parsing network (PSPNet), and UNet showcased MST-DeepLabv3+'s superiority in accuracy and segmentation detail preservation. On the ISPRS dataset, MST-DeepLabv3+ outperformed others in MIoU, OA, precision, recall, and F1-score, with notable gains across all metrics. Results demonstrated its efficacy in delineating specific land cover types, with superior performance in boundary recognition and overall accuracy.

Similarly, on the GID dataset, MST-DeepLabv3+ exhibited superior performance across all evaluation metrics, particularly excelling in MIoU and OA. Detailed classification comparisons highlighted its ability to accurately segment various land cover types, mitigating issues like misclassification and incomplete segmentation observed in other models.

The Taikang cultivated land dataset analysis further confirmed MST-DeepLabv3+'s effectiveness, surpassing PSPNet, UNet, and DeepLabv3+ in MIoU, OA, precision, recall, and F1-score. Visual comparisons showcased its superior boundary delineation and accurate classification of land cover categories.

Ablation experiments demonstrated the progressive improvement of MST-DeepLabv3+ through network modifications, with a notable reduction in model parameter size compared to other models. The compact parameter size contributed to faster training efficiency without compromising segmentation accuracy.

Advancements in Semantic Segmentation

The MST-DeepLabv3+ model addressed challenges in high-resolution remote sensing image segmentation by integrating lightweight networks, attention mechanisms, and transfer learning. By adopting MobileNetV2 as its backbone, it reduced parameter count and enhanced training speed. The addition of SENet in the encoding phase refined feature extraction, compensating for accuracy loss.

Transfer learning further improved segmentation quality by leveraging pre-trained model parameters. Achieving MIoU scores of 82.47%, 73.44%, and 90.77% on diverse datasets, MST-DeepLabv3+ demonstrated superior accuracy in identifying intricate details of remote sensing imagery. Despite its efficacy, the model maintained a compact parameter size of 22.96 megabytes (MB), significantly enhancing training efficiency.

Conclusion

In conclusion, the MST-DeepLabv3+ model presented a breakthrough in high-resolution remote sensing image segmentation. By leveraging MobileNetV2, SENet, and transfer learning, it achieved superior accuracy while reducing computational complexity. With MIoU scores reaching 82.47%, 73.44%, and 90.77% on diverse datasets, it outperformed existing models.

Despite its compact 22.96 MB parameter size, it significantly enhanced training efficiency. Future enhancements may include integrating edge extraction models and multispectral information for further precision and generalization improvements in remote sensing image segmentation.