Unsupervised CycleGAN for Weakly Conductive SEM Samples

By Soham NandiReviewed by Susha Cheriyedath, M.Sc.Mar 27 2024

In an article published in the journal Nature, researchers proposed an unsupervised method based on a cycle-consistent generative adversarial network (CycleGAN) to enhance the quality of scanning electron microscopy (SEM) images for weakly conductive materials.

Traditional methods struggle with weakly conductive samples, but this approach utilized unpaired blurred and clear SEM images for end-to-end learning. Additionally, an edge loss function was introduced to recover finer details in the generated images. The method performed better than traditional techniques, promising significant implications for materials analysis and image restoration.

Background

SEM is a pivotal tool for analyzing submicron-scale structures across various disciplines, including materials science, biomedicine, and chemistry. However, the quality of SEM images heavily relies on sample conductivity, posing challenges for weakly conductive materials commonly encountered in research. Traditional methods, like vacuum sputtering with gold coating, are effective but obscure sample details and incur additional costs.

Furthermore, existing image post-processing techniques, while effective, often require paired data for supervised learning, which is challenging to obtain for weakly conductive samples. Recent advancements in deep learning, particularly CycleGAN, have enabled unsupervised learning approaches, presenting opportunities to overcome these limitations. However, prior applications of CycleGAN in SEM image enhancement primarily focused on supervised learning or lacked consideration for specific material structure analysis needs.

This paper addressed these gaps by proposing an unsupervised approach using CycleGAN to enhance SEM image quality for weakly conductive samples. By leveraging unpaired blurry and clear SEM images, the model learned to perform image deblurring without relying on paired data, thus overcoming the challenge of obtaining such data for weakly conductive samples.

Additionally, introducing an edge loss function tailored to material structure analysis requirements ensured detailed information restoration while eliminating artifacts. The novelty of the study was in its combination of unsupervised learning with CycleGAN and consideration of material structure analysis requirements, offering a promising solution to enhance SEM imaging quality for weakly conductive samples without the need for complex physical operations or paired data.

Principle and network analysis

Inspired by CycleGAN, the proposed method utilized two generators and two discriminators to enhance SEM image quality. Operating on unpaired sets of blurred and clear images, the generators aimed to translate between blurred and clear domains. Adversarial loss drove this translation process, with least squares loss employed to generate high-quality images.

Cycle-consistency loss ensured fidelity to input images, while the structure similarity index measure (SSIM) loss ensured perceptual similarity. Identity loss and edge loss preserved original information and enhanced edge details, respectively. The generators were trained simultaneously to learn mapping relationships between image domains. The architecture comprised a Unet network structure with multi-scale convolution for the generator and a full convolutional approach for the discriminator.

The generator included eight convolution and deconvolution layers, with 14 multi-blocks enhancing edge features. The discriminator employed five convolution layers, followed by instance norm and leaky rectified linear unit (ReLU) activation functions. Stride sizes varied to optimize convolutional operations.

Evaluation results demonstrated the efficacy of the method in deblurring SEM images, even in scenarios with complex blur types. Overall, the proposed framework offered a robust solution for enhancing SEM image quality without the need for paired data, paving the way for advancements in material analysis and image restoration.

Method

The research employed various image quality metrics, including average gradient (AG), contrast (CON), spatial frequency (SF), SSIM, and peak signal-to-noise ratio (PSNR). These metrics quantified image clarity, contrast, and similarity between images. Loss functions like adversarial loss, cycle consistency loss, identity loss, and edge loss were utilized to train the proposed CycleGAN model.

The model was trained on synthetic datasets containing Gaussian blur, synthetic fog, and hybrid blur. Real datasets were also used, comprising blurry SEM images of weakly conductive samples induced by deliberate contamination. The training involved TensorFlow on an NVIDIA GeForce RTX 3090 graphic processing unit (GPU) for 50 epochs with a fixed image size of 256x256 pixels.

Verification and analysis of experimental results

The research presented a method utilizing CycleGAN to enhance SEM image quality, particularly for weakly conductive samples. To quantitatively assess the model, simulated datasets with varying blur types were created. CycleGAN outperformed traditional methods like blind deconvolution and Wiener filtering, especially in complex blur scenarios.

Evaluation metrics such as SSIM and PSNR corroborated the superiority of CycleGAN, showcasing its efficacy in restoring image clarity and contrast. Real dataset evaluations further validated the model's effectiveness across different materials and blur types. CycleGAN consistently produced clearer images with better contrast and detail preservation compared to traditional methods.

Additionally, experiments on weakly conducting materials not trained by the network demonstrated CycleGAN's capability to enhance SEM images beyond contamination-induced blurring. Incorporating edge loss in the network architecture further improved image quality, particularly in preserving edge details crucial in micro-nano scale SEM images. Optimal edge loss weight was determined through quantitative evaluation, and experiments confirmed its effectiveness in reducing artifacts and maintaining edge sharpness.

Overall, the study provided a robust framework for SEM image enhancement, offering significant advancements over traditional methods. The CycleGAN model demonstrated adaptability to various blur types and materials, showcasing its potential for broad applications in material analysis and image restoration in the SEM domain. 

Conclusion

In conclusion, the proposed unsupervised CycleGAN-based method effectively enhanced SEM imaging quality for weakly conductive samples, surpassing traditional techniques. By introducing an edge loss function tailored to material analysis needs, the approach removed artifacts and restored crucial material contour details.

The authors pioneered the application of unsupervised learning in SEM image enhancement, offering significant advancements in material analysis. The method held substantial promise for broadening the scope of artificial intelligence applications in materials science, facilitating more accurate and detailed material analysis processes.