AE-APT: Enhanced Detection of Advanced Cyber Threats

By Muhammad OsamaReviewed by Susha Cheriyedath, M.Sc.Jul 11 2024

In an article recently submitted to the arXiv* server, researchers introduced a novel deep learning approach called AutoEncoders-Advanced Persistent Threats (AE-APT) for detecting APTs in highly imbalanced datasets. Their aim was to enhance cybersecurity defenses by identifying anomalous patterns indicative of APT activity.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

Background

The growing use of information technology (IT) in activities like e-commerce, asset sharing, and handling private data has made cybersecurity very important. APTs are a big risk to organizations, governments, and individuals because they involve complex cyberattacks that aim to access systems without permission and stay hidden for long periods. These attacks are carefully planned to avoid existing security measures, often targeting specific groups to steal sensitive data, harm important systems, or cause disruption.

Traditional methods for detecting APTs rely on anomaly detection-based security techniques, which often struggle to effectively identify these hidden attacks. Detecting APTs is particularly challenging due to the scarcity of relevant data and the imbalance in the data. This difficulty highlights the need for more advanced methods.

About the Research

In this paper, the authors proposed AE-APT, a deep learning-based tool for APT detection that uses AE methods. AE-APT consists of six neural networks: a Baseline and five variations, including adversarial AE (AAE), recurrent neural network AE (RNNAE), long short-term memory AE (LSTMAE), gated recurrent unit AE (GRUAE), and attention-based AE (ATAE).

These models were trained on data labeled as normal system activity, learning to represent typical system behavior in a low-dimensional space. Any data that deviated significantly from this representation during the decoding process was flagged as anomalous, potentially indicating an APT attack.

The researchers used an ensemble learning mechanism, combining the anomaly scores from each AE model using a majority aggregation technique. This approach leverages the strengths of different architectures to enhance overall detection accuracy and robustness.

Furthermore, AE-APT was evaluated on a suite of provenance trace databases produced by the DARPA Transparent Computing program, which includes data from multiple operating systems, such as Android, Linux, BSD, and Windows. The datasets cover two attack scenarios, with APT-like attacks constituting as little as 0.004% of the data.

Research Findings

The outcomes showed that AE-APT achieved significantly higher detection rates compared to its competitors: valid frequent association rule mining (VF-ARM), valid rare association rule mining (VR-ARM), attribute value frequency (AVF), frequent pattern outlier factor (FPOF), outlier degree (OD), and one-class classification by compression (OC3). The ATAE model, especially, showed exceptional effectiveness, achieving normalized discounted cumulative gain (nDCG) scores ranging from 0.7 to 0.92 across various operating systems and attack scenarios. This underscored the attention mechanism's crucial role in improving anomaly detection by enabling the model to focus on critical data segments.

The authors observed that recurrent models like RNNAE, LSTMAE, and GRUAE performed well on extensive datasets with longer event sequences, indicating their capability to capture long-term dependencies in system behavior. They also presented visual rankings of anomalous processes by the baseline AE and AAE models, illustrating AE-APT's efficacy in detecting and isolating APT-related activities within the datasets.

Applications

This research has significant implications for enhancing cybersecurity defenses against APTs. AE-APT serves as a powerful tool for organizations, governments, and cybersecurity experts to detect and mitigate these threats. Its capability to accurately identify and isolate APTs across diverse operating systems makes it a versatile and robust solution against the evolving threat landscape.

The explainability and interpretability of AE-APT are important for practical cybersecurity applications. Its visualization tools offer clear representations of anomalies, helping cybersecurity professionals comprehend the potential impact of identified threats. This clarity supports informed decision-making and effective response strategies. The integration of attention mechanisms within the transformer-based AE architecture further enhances the model's ability to detect subtle anomalies.

Conclusion

In summary, the novel approach effectively detected APT-like activities across multiple operating systems, demonstrating deep learning's potential in addressing complex cybersecurity threats. Its success with diverse and challenging datasets underscored the efficiency of these techniques.

Moving forward, the researchers proposed refining AE-APT to handle more complex threats. This involves exploring new deep learning models, integrating expert feedback, and implementing online learning for real-time analysis and response. Enhancing the framework's interpretability and explainability through additional visualization tools was also recommended to improve communication between human experts and AI. By advancing in these areas, AE-APT can evolve into a more powerful tool for protecting systems against the continuously evolving threat landscape.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.