A Holistic review and performance evaluation of unsupervised learning methods for network anomaly detection

Various studies have explored unsupervised AD algorithms for IDS. In 2000, one study compared cluster-based estimation, KNN, and one-class SVM using the KDD Cup 99 dataset [17]. In another analysis in 2003, the authors evaluated AD approaches on the DARPA98 dataset, showing promising results with local outlier factor (LOF) and unsupervised SVM [18]. In 2007, the authors reviewed different clustering schemes and analyzed that K-harmonic means performed well in speed, while population-based stochastic techniques were suitable for cluster quality [19]. Additionally, in a study in 2007, seven distinct IDS techniques for HTTP anomalies were tested and compared under simulated and real-world settings. It was found that the token-based approaches exhibited better discrimination and generalization using heuristics-based approaches [20].

In 2009, the authors presented literature studies to review machine learning (ML) methods for IDS, including single, hybrid, and ensemble classifiers [21]. In 2012, the authors reviewed and compared different clustering techniques based on parameters such as data type, cluster type, complexity, dataset, and measures, along with their advantages and disadvantages [22]. In another study in 2014, the authors present an overview and comparison of four novelty detection techniques. The experimental outcomes indicate that the KNN novelty detection scheme performs better than other methods [23]. Moreover, in 2015, the authors provided a comprehensive review of the 19 most popular clustering algorithms with a detailed analysis of traditional and modern approaches to emphasize the importance of the essential data-analysis technique, clustering [24].

In 2016, a comprehensive assessment of 19 unsupervised AD techniques was conducted on 10 datasets, showcasing their practical significance across diverse applications [1]. Another study in 2016 compared four AD algorithms, highlighting the superior performance of one-class SVM in smart city wireless sensor networks [2]. In 2018, a survey focused on comparing feature techniques and bridging the gap between IDS and forensic investigation [3]. A 2019 study evaluated unsupervised AD algorithms against unified attack models and addressed research questions related to performance. A detailed review in 2019 explored UL methods in various networking domains, discussing recent advances, challenges, and future directions [4]. A general introduction to the ML algorithms applied to IDS can be found in (2020). The proposed work also performs experiments on the KDD-CUP dataset to evaluate the performance of ML algorithms [5]. In 2021, the authors presented a comprehensive survey on IDS with special attention to feature-selection methods, different types of datasets, challenges, and possible future directions [6].

The survey conducted in 2021 categorizes anomaly detection algorithms into seven distinct categories based on their working mechanisms [7]. These categories include statistical models, density-based methods, distance-based techniques, clustering approaches, isolation methods, ensemble methods, and subspace techniques. For each category, the survey provides insights into the time complexity of the algorithms and outlines their general advantages and disadvantages. Furthermore, a detailed and systematic survey of clustering methods is provided in (2022). This paper covers an in-depth overview of state-of-the-art clustering methods, performance, open issues, and various application areas of clustering methods [8]. A comprehensive overview of UL techniques for AD was conducted in 2022, emphasizing their utility in scenarios with scarce labeled data [9]. It highlights the challenges associated with traditional outlier detection methods and underscores the potential of unsupervised approaches in handling raw data effectively. Another survey was conducted in 2022 to delve into unsupervised anomaly localization (AL) using DL for industrial image inspection [10]. With its in-depth technical insights, the survey emerges as a vital resource for researchers venturing into industrial AL and seeking to extend its applications across diverse domains.

In a 2023 survey, outlier detection strategies from seven families were analyzed, revealing that consensus-based strategies derived from DL outperformed standalone UL methods [11]. Another comprehensive study in 2023 explored anomaly detection techniques within sensor networks and the internet of things (IoT) [12]. It unveiled a significant focus on detecting malicious activities, spanning various ML and DL approaches, with a notable surge in interest between 2017 and 2022. The study showcased a significant adoption of DL and ML techniques, notably MLPs and graph-based methods, across various anomaly detection applications, reflecting the evolving landscape of IoT security measures and the need to address emerging threats comprehensively. A review conducted in 2023 focused on different clustering categories within semisupervised and UL approaches [13]. Furthermore, another survey conducted in 2023 delved into automatic log file analysis for early incident detection, particularly focusing on self-learning anomaly detection techniques [14]. These methods, predominantly leveraging DL neural networks, demonstrated superior performance over traditional approaches and addressed issues with unstable data formats. A comparison of existing anomaly detection surveys with our study is presented in Table 1.

From Table 1, existing literature offers surveys on ML techniques for NAD and other domains but lacks a comprehensive assessment of UL techniques specifically for NAD systems. This paper fills that gap by comparing UL techniques based on relevant parameters, as mentioned in Table 1. It also includes recent advances in unsupervised NAD and empirical evaluations on benchmark datasets to analyze performance across conventional and modern datasets.

An unsupervised approach based on density and grid-based clustering was proposed in 2005 [46]. It effectively classified a high-dimensional dataset into a collection of clusters in which the points not associated with any cluster are flagged as abnormal. The presented strategy achieved comparable outcomes; still, the FPR was high [46]. Another advanced method based on fuzzy rough C-means clustering was introduced in further research [47, 48]. K-means clustering is a well-known methodology for finding anomalies, outperforming prior unsupervised algorithms in accuracy [49]. Later 2012, a new variant was introduced that merged K-means clustering with the C4.5 decision tree approaches to generate more effective outcomes than previous methods [50]. Density-based clustering methods excel at identifying clusters of arbitrary shapes, while grid-based clustering schemes offer fast processing.

Leung and Leckie [46] proposed the merging of adaptive intervals approach to spatial data (MAFIA). This novel mechanism combines density-based and grid-based clustering to discover high-dimensional input space clusters. Gujral et al. [51] employed spectral clustering to identify scattered anomalies and form representative clusters. A sparse symmetric matrix was utilized with a KNN to address scalability in IDS. In another work [52], a two-fold framework for NAD in high-dimensional data was proposed by Ni et al. [52] The framework uses a clustering technique to approximate cluster centers and the mutual information coefficient to remove redundant and irrelevant features. Comparing experimental results to those of other unsupervised approaches, accuracy was enhanced. To represent normal data into clusters, Bhuyan et al. [53] presented an effective outlier-based technique employing a variation of k-means clustering. Based on outlier scores that exceed a threshold, anomalies are found.

Syarif et al. [54] compared misuse detection and AD algorithms using four classifiers (Naïve Bayes, rule induction, NN, and decision tree) for misuse detection and five clustering algorithms (k-means, k-medoids, EM clustering, and distance-based outlier detection) for AD. Evaluation of DARPA98 and NSL-KDD datasets showed that detecting unknown attacks remains challenging for misuse detection. Rule induction achieved 99.58% accuracy with a 0.40% FPR for known attacks, while the decision tree classifier performed poorly with 63.97% accuracy and a 17.90% FPR. In contrast, the distance-based outlier detection algorithm achieved 80.15% accuracy in detecting unknown intrusions in the AD module. Bhuyan et al. [55] proposed Tree-based Subspace Clustering (TreeCLUS), a UL approach for detecting network anomalies with a minimized FPR. It utilizes cluster stability analysis to identify effective clusters and a multi-objective scheme for cluster labeling.

In another work, Prasad et al. [56] proposed an unsupervised feature selection technique for IDS. This technique selects the most informative features by calculating feature scores using entropy and computing the feature histogram. The study also introduced a novel cluster center initialization approach to address the limitations of conventional clustering techniques. The approach computed semi-identical instances, created sets, and avoided outliers as initial cluster centers. Micro-clusters were formed based on spatial distance, and similar micro-clusters were merged into macro-clusters. The presented cluster centralization approach outperformed basic clustering methods, requiring fewer iterations to obtain final clusters and improving the accuracy of the detection model.

UNADA [57] is a distributed computing framework-based scheme designed to enhance the detection of NAD unsupervised algorithms. It leverages a density-based grid algorithm deployed on a large cluster of servers to process high traffic volumes efficiently. However, DBSCAN's limitations in handling high-dimensional data are addressed by introducing subspace clustering and evidence accumulation techniques for parallel processing across multiple servers. Despite its advantages, UNADA's time complexity hampers its suitability for real-time and scalable anomaly detection. In another work [58], a real-time UL anomaly detection scheme for streaming data is proposed. Hierarchical temporary memory (HTM) detects spatial and temporal anomalies in noisy environments. The strategy adapts well to changing data streams while minimizing false positives. However, there is potential for further accuracy improvement by adopting ensemble-based approaches. Additionally, the exploration of real-time anomaly detection techniques for multivariate data streams is crucial to address the challenges of the current scenario.

SOMs are frequently applied as unsupervised tools for identifying malicious and abnormal network behavior. The characteristic of SOMs is that they can automatically organize and infer patterns from a range of inputs and then assess whether a piece of new information conforms to the deduced pattern or not, hence identifying anomalous inputs [6, 59, 60]. Landress [61] proposed a fusion model for UL in intrusion detection to address false positives. The model combines K-means clustering, J48 decision tree, and SOM using the KDDCUP99 dataset. Clustering is performed with k-means, grouping attacks into normal, Neptune, smurf, and ipsweep categories. J48 decision tree is then used for feature selection, reducing false positives by selecting optimal features. SOM is applied to visualize the data and categorize abnormal behavior.

Huang and Huang [62] employed GHSOM to analyze anomalous traffic behavior by analyzing node density and samples. The network traffic data are analyzed via visual representation of attack patterns concerning hierarchical relationships. In the detection phase, GHSOM is combined with SVM, which provides classification rules to identify the network traffic into normal and abnormal behaviors.

PCA is commonly used for dimensionality reduction in IDS [63, 64], but ICA is proposed as an alternative for non-Gaussian data [65]. Pattewar and Sonawane [66] present a feature extraction approach based on PCA and KPCA. In the first approach, PCA is utilized for extracting features. KPCA and Bayesian methods are employed in the second approach to eliminate the noisy features, thereby removing the critical features that result in better accuracy and computational efficiency than that of PCA.

In another work proposed by Kuang et al. [67], hybridization of the KPCA, GA, and SVM model is employed. In the proposed scheme, KPCA is applied to elicit the core features and the NRBF kernel is employed to lessen the training time of SVM. At the same time, GA optimizes the parameters of SVM to avoid overfitting. The experimental findings reveal that the proposed approach shows better generalization performance and a higher accuracy rate than the SVM classifier. In the work of Elkhadir et al. [68], a comparative analysis of PCA and KPCA is performed on the KDDCUP99 dataset. Experiments reveal that when the number of NN is between one and four, a KPCA with a quadratic kernel offers a higher detection rate (DR) than a PCA, notably for identifying DOS and PROBE attacks.

Unlike attack records, normal connection records are often readily available in real-world applications. As a result, one-class SVM trained on a dataset consisting of normal connection records can effectively differentiate between normal and varied attacks. Amer et al. [69] report that one-class SVM is susceptible to data outliers and suggest two modifications: robust one-class SVM and ETA one-class SVM. The central concept for robust one-class SVM is to minimize the mean square error (MSE) for dealing with outliers by utilizing the class mean as the averaged information. In contrast, ETA one-class SVM employs an outlier suppression mechanism. The proposed approach indicated that the enhanced one-class SVM performed better on two of four datasets than the existing UL AD methods.

Nguyen et al. [70] further improve the performance of one-class SVM by introducing the nested one-class SVM. Radial basis function (RBF) or a Gaussian kernel is commonly used to kernelize one-class SVM. However, selecting a Gaussian kernel hyperparameter based on the attack's availability is impractical. The authors attempted to solve the problem of kernel parameter optimization by proposing data-driven hyperparameter optimization. The presented methodology has achieved high detection accuracy and a lower false alarm rate (FAR) than the original one-class SVM in the experiments.

Verkerken [71] conducted a study on the generalization ability of UL algorithms in NAD. The four UL methods used for evaluation were PCA, IF, autoencoder (AE), and one-class SVM. The models were trained on one dataset and evaluated on another to assess their inter-dataset generalization. Experimental results showed that one-class SVM achieved the best AUROC scores, followed by the AE and PCA. However, when evaluated on different datasets, there was a significant drop in performance for all models, indicating a lack of generalization. The inter-dataset evaluation scheme highlighted the low generalization strength of the models. The study suggests that further research is needed to improve the adaptability and generalization of UL models for real-time detection in practical applications.

Zoppi et al. [26] conducted an extensive study investigating the relationship between attack families, anomaly classes, and detection methods. Their experiments on various UL algorithms, including ODIN, K-means, LOF, FastABOD, HBOS, and one-class SVM showed that FastABOD is effective for point anomalies, K-means performs well for collective anomalies, and HBOS is lightweight and efficient for limited computing resources. Additionally, the analysis of HBOS in Ref. [72] found that dynamic bins are particularly useful for identifying R2L and U2R attacks. NN-based approaches are commonly used and exhibit strong performance due to their unsupervised nature and outlier detection criteria. However, they may face challenges with high-dimensional data and incur computational complexity when dealing with large datasets. On the contrary, SVM maximizes the machine's accuracy concerning the training dataset while maximizing its capacity to correctly classify subsequent testing datasets, even on high-dimensional datasets. Therefore, combining NN-based approaches and SVM enhances classification performance with reasonable computational cost in high-dimensional data [73]. In contrast, SVMs maximize accuracy on training and testing data, including high-dimensional datasets, with one-class SVMs focusing on discovering decision boundaries with maximum distance from the origin [73].

Falcão et al. [4] compared various UL algorithms for NAD. Classification algorithms were found to exhibit the highest accuracy among the methods considered. Angle-based algorithms showed robustness in parameter selection. However, detecting anomalies with nonrepeatable and unstable behavior remained challenging, and higher detection scores were observed in datasets with rare anomaly events. In the literature, many proposed NAD schemes rely on traditional distance metrics such as Euclidean or Mahalanobis distance, which may not adequately capture the heterogeneous nature of network features.

To overcome this limitation, Aliakbarisani et al. [74] proposed a linear metric learning approach incorporating side information and UL techniques to extract similarity or dissimilarity constraints. Their approach formulates an optimization problem using the constraint trace ratio and utilizes Laplacian Eigenmap for preserving local neighborhood information. Experimental results on Kyoto 2006+ and NSL-KDD datasets demonstrated improved accuracy and reduced FPRs compared to Euclidean-based distance metrics.

Preprocessing steps are essential in NAD. One common technique is using LOF to remove outliers from the training set. LOF detects network anomalies by comparing them with new data and estimating the LOF value based on preprocessed normal datasets [75]. Additionally, Ding et al. [76] proposed hybridizing the similarity matrix scheme and LOF as preprocessing steps to identify outliers by considering data objects with lower similarity than others. This approach enhances outlier detection adaptability, especially for datasets with irregular shapes.

With the rapid growth of the IoT, attackers are exploiting vulnerabilities in insecure devices like IP cameras to launch devastating Distributed Denial of Service (DdoS) attacks [77]. Traditional defense methods that rely on predefined features extracted from predefined flows are often ineffective in promptly detecting and blocking malicious flows. Hwang et al. [83] proposed a new scheme for automatically profiling traffic patterns directly from raw traffic to address this. By examining the initial packets and utilizing a convolutional neural network (CNN), the method generates features from raw traffic, enabling auto-learning. An unsupervised AE is then used to create profiles of benign traffic, effectively filtering out abnormal traffic. Experimental results showed a promising outcome with 100% accuracy and low false negative (0.02%) and false positive (0.83%) rates achieved by the proposed method.

Kabir and Luo [78] evaluated the performance of traditional and DL-based unsupervised methods for network flow anomaly detection. The study compared SOM, K-means, deep autoencoding GMM (DAGMM), and adversarially learned anomaly detection ALAD algorithms. Experimental results on the KDDCUP99 dataset showed that DAGMM achieved the lowest FPR (0.9%), while SOM had the highest accuracy (99.9%). SOM and k-means were found to be easy to implement with stable performance. ALAD performed well in detecting minority attacks due to generating adversarial samples.

In another study, Truong-Huu et al. [79] investigated the deployment of generative adversarial networks (GANs) for NAD. They utilized the UL scheme, which does not require prior label information during training. The concepts of traffic sessions and traffic aggregation techniques were introduced to extract useful traffic features and learn traffic behaviors. The experiments conducted on UNSW-NB15, CIC-IDS2017, and synthetic datasets demonstrated that GAN outperformed existing anomaly detection approaches.

Sovilj et al. [80] addressed the lack of descriptive analysis in intrusion detection models by introducing an attentional model analysis for DL models. This analysis provides explanations and predictions, aiding network operators in understanding potential threats. The authors conducted a comparative analysis of deep neural learning architectures for IDS in sequential data streams. Experimental results on synthetic and CIC-IDS2017 datasets showed that the RNN variant outperformed the nonsequential deep AE for anomaly detection. However, the performance of the attentional model could be further improved by incorporating attentional components to detect anomalous input data segments. Another study by Carrera et al. [81] addresses the critical need for detecting zero-day cyber-attacks, which exploit undisclosed vulnerabilities, posing substantial technical and economic risks to enterprises and technology-dependent systems. Through benchmarking three different approaches, such as deep autoencoding with GMM and IF, the proposed work demonstrates enhanced accuracy with IF. Furthermore, employing the SHAP methodology elucidates crucial features for attack classification, as validated through experiments conducted on diverse datasets, including KDD99, NSLKDD, and CIC-IDS2017. Sáez-de-Cámara et al. [82] address IoT security challenges by proposing a federated learning (FL) architecture for unsupervised network intrusion detection. Leveraging FL mitigates network overhead and data isolation issues, enhanced by an unsupervised device clustering algorithm. Evaluation on a diverse testbed showcases successful device grouping, although clustering quality may diminish with insufficient local training data. Notably, FL-based models exhibit faster convergence and low FPRs, outperforming ML-based alternatives like Kitsune, signaling promise for real-world deployment. However, challenges remain in adapting the approach to high-mobility IoT settings and addressing compromised devices in adversarial scenarios, warranting further exploration. The comparison analysis of UL approaches employed for network anomaly/intrusion detection is summarized in Table 2, providing an overview of the findings and insights regarding the performance and effectiveness of different methods.

Table 2 offers a comprehensive overview of various network anomaly and intrusion detection strategies. It delineates different methodologies and algorithms, ranging from subspace clustering to tree-based clustering and optimization techniques, all tailored to different datasets. The performance metrics exhibited by these approaches vary significantly, with some achieving commendable accuracy and DRs.

The table delineates these methodologies and offers a nuanced insight into their performance metrics. Some methods demonstrate impressive accuracy and DRs, showcasing their efficacy in identifying anomalous activities within network traffic. However, a notable observation is the need for more emphasis on real-time detection capability across these methodologies. Real-time detection is imperative for promptly identifying and mitigating security threats, yet not all approaches prioritize this aspect adequately.