Enhancing Few-Shot Semantic Segmentation with Transductive Meta-Learning

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Feb 22 2024

In a paper published in the journal Scientific Reports, researchers introduced a novel method for few-shot semantic segmentation, addressing critical challenges in performance. They proposed leveraging an ensemble of visual features learned from pre-trained classification and semantic segmentation networks with the same architecture, effectively enhancing informativeness.

This approach used a pre-trained semantic segmentation network to mitigate false positives during inference and employed transductive meta-learning for improved prediction in cases of poor similarity between support and query images. Experimental results on benchmark datasets showed significant performance improvement with minimal trainable parameters, achieving state-of-the-art results on various datasets.

Related Work

Past work in deep neural networks demonstrated their capability to learn rich visual features from large labeled datasets, benefiting critical applications like medical imaging. However, when faced with limited labeled examples, the generalization ability diminishes, especially in domains like geospatial and medical imaging, where data collection is challenging.

In response to this challenge, few-shot learning seeks to emulate the rapid learning abilities observed in humans by utilizing only a limited number of examples. This approach holds promise for applications in diverse fields where data scarcity is a significant challenge, potentially unlocking new opportunities for machine learning in various domains.

Methodology: Two Passes, Comparison, Voting

The methodology described involves a two-pass approach to learning intra-class and intra-object similarity for few-shot semantic segmentation. In the first pass, the model utilizes support and query images to actively learn intra-class similarity by comparing features from the support with visually similar features in the query.

A frozen pre-trained classification network is a backbone, learning features for the support and query images. These features encode spatial distribution and shape information, supplemented by features learned by a frozen semantic segmentation network trained on background and base classes.

Support features from both backbones are processed to remove background-related features, and cosine similarity is computed between support and query features at different depths, generating multi-scale 4D volumes. 4D convolutions are applied to these volumes, followed by processing to reduce dimensions and generate segmentation masks through decoders. The process concludes with a comprehensive loss function, including a transductive loss term to reduce false positives.

In the second pass, the query image serves as both support and query, aiming to learn intra-object similarity by propagating visually similar features identified in the first pass. This pass employs a semantic segmentation backbone, with features extracted and processed similarly to the first pass.

A decoder maps correlations into segmentation maps supervised by loss functions. Researchers train the proposed meta-learner using episodic training, assigning equal weights to each loss term. They implement an extension to the K-shot setting, involving multiple forward passes and a voting mechanism to predict segmentation masks, enhancing the model's performance in handling few-shot scenarios.

Experimental Setup and Analysis

The experiments encompass implementation details, training procedures, evaluation metrics, and ablation studies. In terms of modules, the backbones utilized are frozen residual network (ResNet)-)-style architectures pre-trained through supervised learning on imageNet-1K and supervised segmentation learning on base classes specific to each fold. Additionally, researchers employ 4D convolutions with a uniform architecture and weights, totaling 2.5M trainable parameters. Two decoders sharing the same architecture are utilized alongside episodic training for meta-learning, utilizing the frozen backbones.

Training involves a two-phase approach: pretraining and meta-training. A supervised segmentation model is initially trained on base classes associated with each fold using a pyramid scene parsing network (PSPNet) with ResNet50 and ResNet101 backbones. It is followed by meta-training the entire model with frozen backbones using episodic learning. Batch sizes and optimizer settings vary between the Pascal- and common objects in context (COCO) datasets, with training conducted on four NVIDIA V100 graphics processing units (GPUs).

Researchers perform evaluations on benchmark datasets, namely Pascal and COCO, partitioned into four folds. Mean intersection-over-union (mIoU) is the primary evaluation metric, reporting results on individual folds and averages across all folds for both datasets.

Results indicate state-of-the-art performance for both 1-shot and 5-shot settings, particularly with a Resnet-101 backbone. Comparisons with existing methods, such as Min et al., demonstrate superior performance. Additionally, qualitative comparisons showcase the method's effectiveness in handling challenging scenarios, as evidenced by visual results.

Ablation studies further validate the method's efficacy, demonstrating substantial performance improvements over baseline methods. Experiments involving different backbone configurations, including single and dual backbones, highlight the importance of backbone diversity in enhancing performance. The method's adaptability to different backbone combinations is showcased through performance gains in various experimental setups, emphasizing its robustness and versatility in few-shot semantic segmentation tasks. Additional details on experiments are available in the supplementary information.

Conclusion

To sum up, researchers introduced a novel two-pass end-to-end approach for few-shot semantic segmentation, tackling key performance challenges. The method uses pre-trained classification and semantic segmentation networks to capture diverse information at various depths and reduce false positives. The first pass matches support and query features to address intra-class similarity, while the second pass suppresses false positives and propagates query features for intra-object similarity. Experimental results show significant performance gains on benchmark datasets, particularly state-of-the-art results with Resnet-101 on both Pascal- and COCO-20.