Self-Supervised Learning Boosts Sewer Anomaly Detection With Better Accuracy

By Muhammad OsamaReviewed by Joel ScanlonOct 10 2024

New research reveals that advanced self-supervised learning models, such as SimCLR and Barlow Twins, can significantly improve anomaly detection in sewer systems, even when defect data is scarce—paving the way for smarter infrastructure inspections.

Research: Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methods. Image Credit: SvedOliver / Shutterstock

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

A research paper recently posted on the arXiv preprint* server comprehensively explored applying advanced machine learning techniques, particularly self-supervised learning (SSL), for anomaly detection in vision-based infrastructure inspection. SSL methods were rigorously evaluated for their robustness in real-world scenarios, focusing on extreme cases of class imbalance.

The researchers focused on sewer infrastructure, utilizing the Sewer-ML dataset containing 1.3 million images from video inspections. They conducted various binary classification experiments to evaluate the performance of different SSL methods under multiple levels of class imbalance. These experiments were designed to systematically assess defect proportions of 1%, 2%, 5%, and 15%, simulating real-world conditions where labeled defect data is scarce. The study highlighted SSL's potential to improve anomaly detection and open pathways for further research in this field.

SSL in Anomaly Detection

SSL is a machine-learning approach that involves training models using unlabeled data. This is achieved by generating surrogate labels through tasks that utilize the inherent patterns or structures within the data. These pretext tasks enable the model to learn rich representations that can be generalized across various downstream tasks without needing manual labeling. This method allows the model to learn useful representations that can be used for different tasks without needing manual labeling.

This technique is particularly useful for semi-supervised anomaly detection tasks, especially when acquiring labeled data, which can be expensive, time-intensive, or difficult. However, despite these advantages, anomaly detection remains underrepresented in SSL research, as large-scale studies frequently overlook its inclusion in benchmarks. This gap is particularly concerning in scenarios where imbalanced datasets, such as sewer inspection data, are prevalent. Therefore, exploring SSL's potential for improving anomaly detection models is crucial, mainly for applications like infrastructure inspection, where early detection of anomalies can prevent costly failures.

Evaluating SSL for Sewer Infrastructure

In this paper, the authors address the gap in SSL research on anomaly detection. They conduct a comprehensive evaluation of SSL methods for real-world anomaly detection, focusing on sewer infrastructure. The Sewer-ML dataset, a multi-label dataset featuring 17 defect classes, was used to rigorously assess the performance of lightweight models, including Vision Transformer-Tiny (ViT-Tiny) and Residual Network-18 (ResNet-18), across various SSL frameworks.

The Sewer-ML dataset, featuring 17 defect classes, evaluated lightweight models like Vision Transformer-Tiny (ViT-Tiny) and Residual Network-18 (ResNet-18). These models were trained and tested under varying proportions of defect samples, with training involving defect proportions as low as 1% to simulate extreme real-world class imbalance. These models were tested across various SSL frameworks, including Bootstrap Your Own Latent (BYOL), Barlow Twins, Simple Framework for Contrastive Learning of Visual Representations (SimCLR), Distillation with No Labels (DINO), and Masked Autoencoders (MAE). Particular emphasis was placed on understanding how well these models generalize to unseen data, especially in highly imbalanced scenarios.

The study aimed to determine the effectiveness of SSL methods in handling class imbalance. To achieve this, the researchers conducted 250 experiments with varying proportions of defect samples (1%, 2%, 5%, and 15%) in the training and validation datasets. The performance of each method was evaluated based on the F1 score for both defect and non-defect classifications, with additional metrics like the weighted F2 score (F2CIW), which assigns greater importance to defect recall due to its economic significance.

Experimental Methodology

The methodology involved a preliminary search for hyperparameters to refine the data augmentation process for the dataset. This search focused on optimizing image resolution and adjusting the augmentation parameters to balance model effectiveness with computational efficiency. This step focused on adjusting augmentation parameters and image resolution to achieve stability between the model's effectiveness and computational efficiency.

The authors also conducted an ablation analysis to evaluate anomaly detection using SSL methods, emphasizing their robustness against distribution imbalances. They primarily focused on joint-embedding architectures, which create an embedding space by synchronizing representations of various augmented views of the same input without collapsing. Joint-embedding methods such as Barlow Twins and SimCLR use contrastive learning to maximize the agreement between different augmented versions of the same image without requiring negative samples. Additionally, MAE, a self-supervised technique that fills in missing sections of the input data, was evaluated to determine its effectiveness in handling complex data patterns and improving anomaly detection performance.

Key Findings and Insights

The outcomes highlighted the superiority of joint-embedding methods, such as SimCLR and Barlow Twins, over reconstruction-based approaches like MAE, which struggle with performance under class imbalance. SimCLR, in particular, performed best with ResNet-18 when defect proportions were above 5%, whereas BYOL, which does not rely on negative sampling, excelled in handling extreme imbalance scenarios, such as with 1% defect data. BYOL and ResNet-18 excelled under extreme imbalance levels (1%). This success is likely due to BYOL's training dynamics, which do not require explicit negative sampling.

Barlow Twins demonstrated strong performance in highly imbalanced scenarios by employing a technique that minimizes redundancy, thus preventing overfitting to the prevalent class. In contrast, DINO exhibited an interesting pattern, showing weaker results at 1% and 2% levels when paired with ResNet-18 but not ViT-Tiny. This difference may be linked to DINO's ability to learn global features through knowledge distillation, which pairs better with ViT-Tiny's attention-based architecture.

The study found that the choice of SSL model is more important than the backbone architecture. Neither ViT-Tiny nor ResNet-18 consistently outperformed the other in all scenarios. This finding challenges the common assumption that backbone architecture significantly influences model performance in SSL tasks. Additionally, the authors emphasized the need for better label-free assessments of SSL representations since current methods, like RankMe, do not adequately evaluate representation quality, making cross-validation without labels infeasible.

Applications

This research has significant implications for infrastructure inspection, particularly in enhancing the accuracy and efficiency of anomaly detection in sewer systems. By leveraging SSL methods, defects such as blockages, leaks, and structural weaknesses can be detected more accurately. These improvements in early detection could have substantial economic benefits by preventing infrastructure failures before they become critical. This improvement can help prevent costly failures and enhance safety.

The findings also pave the way for further research into applying SSL to other real-world anomaly detection scenarios. This is especially relevant in cases where labeled data is scarce or difficult to obtain. Future studies may explore adapting SSL techniques to domains beyond sewer inspection, such as medical imaging or industrial fault detection.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.