<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and have been shown to be able to detect anomalies in many practical problems. However, in high dimensional data, conventional outlier detection methods often behave unexpectedly due to a phenomenon called the curse of dimensionality. In this paper, we compare and analyze outlier detection performance in various experimental settings, focusing on text data with dimensions typically in the tens of thousands. Experimental setups were simulated to compare the performance of outlier detection methods in unsupervised versus semi-supervised mode and uni-modal versus multi-modal data distributions. The performance of outlier detection methods based on dimension reduction is compared, and a discussion on using k-NN distance in high dimensional data is also provided. Analysis through experimental comparison in various environments can provide insights into the application of outlier detection methods in high dimensional data.</p>
</abstract>

Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and have been shown to be able to detect anomalies in many practical problems. However, in high dimensional data, conventional outlier detection methods often behave unexpectedly due to a phenomenon called the curse of dimensionality. In this paper, we compare and analyze outlier detection performance in various experimental settings, focusing on text data with dimensions typically in the tens of thousands. Experimental setups were simulated to compare the performance of outlier detection methods in unsupervised versus semi-supervised mode and uni-modal versus multi-modal data distributions. The performance of outlier detection methods based on dimension reduction is compared, and a discussion on using k-NN distance in high dimensional data is also provided. Analysis through experimental comparison in various environments can provide insights into the application of outlier detection methods in high dimensional data.

A Comparative Study for Outlier Detection Methods in High Dimensional Text Data

Department of Computer Science and Engineering

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

{"article-title":"A Comparative Study for Outlier Detection Methods in High Dimensional Text Data"}

Outlier detection aims to find a data sample that is significantly different from other data samples. Various outlier detection methods have been proposed and...

A Comparative Study for Outlier Detection Methods in High Dimensional Text Data

Published Online: Nov 28, 2022

Page range: 5 - 17

Received: Jun 22, 2022

Accepted: Oct 19, 2022

DOI: https://doi.org/10.2478/jaiscr-2023-0001

Keywords
Curse of dimensionality, Dimension reduction, High dimensional text data, Outlier detection

© 2023 Cheong Hee Park, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

A Comparative Study for Outlier Detection Methods in High Dimensional Text Data

Published Online: Nov 28, 2022

Page range: 5 - 17

Received: Jun 22, 2022

Accepted: Oct 19, 2022

DOI: https://doi.org/10.2478/jaiscr-2023-0001

KeywordsCurse of dimensionality, Dimension reduction, High dimensional text data, Outlier detection

© 2023 Cheong Hee Park, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
Curse of dimensionality, Dimension reduction, High dimensional text data, Outlier detection