<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Feature selection is an essential pre-processing step in data mining. It aims at identifying the highly predictive feature subset out of a large set of candidate features. Several approaches for feature selection have been proposed in the literature. Random Forests (RF) are among the most used machine learning algorithms not just for their excellent prediction accuracy but also for their ability to select informative variables with their associated variable importance measures. Sometimes RF model over-fits on noisy features, which lead to choosing the noisy features as the informative variables and eliminating the significant ones. Whereas, eliminating and preventing those noisy features first, the low ranked features may become more important. In this study we propose a new variant of RF that provides unbiased variable selection where a noisy feature trick is used to address this problem. First, we add a noisy feature to a dataset. Second, the noisy feature is used as a stopping criterion. If the noisy feature is selected as the best splitting feature, then we stop the creation process because at this level, the model starts to over-fit on the noisy features. Finally, the best subset of features is selected out of the best-ranked feature regarding the Geni impurity of this new variant of RF. To test the validity and the effectiveness of the proposed method, we compare it with RF variable importance measure using eleven benchmarking datasets.</p>
</abstract>

Feature selection is an essential pre-processing step in data mining. It aims at identifying the highly predictive feature subset out of a large set of candidate features. Several approaches for feature selection have been proposed in the literature. Random Forests (RF) are among the most used machine learning algorithms not just for their excellent prediction accuracy but also for their ability to select informative variables with their associated variable importance measures. Sometimes RF model over-fits on noisy features, which lead to choosing the noisy features as the informative variables and eliminating the significant ones. Whereas, eliminating and preventing those noisy features first, the low ranked features may become more important. In this study we propose a new variant of RF that provides unbiased variable selection where a noisy feature trick is used to address this problem. First, we add a noisy feature to a dataset. Second, the noisy feature is used as a stopping criterion. If the noisy feature is selected as the best splitting feature, then we stop the creation process because at this level, the model starts to over-fit on the noisy features. Finally, the best subset of features is selected out of the best-ranked feature regarding the Geni impurity of this new variant of RF. To test the validity and the effectiveness of the proposed method, we compare it with RF variable importance measure using eleven benchmarking datasets.

A New Noisy Random Forest Based Method for Feature Selection

Cybernetics and Information Technologies

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"A New Noisy Random Forest Based Method for Feature Selection"}

Feature selection is an essential pre-processing step in data mining. It aims at identifying the highly predictive feature subset out of a large set of...

A New Noisy Random Forest Based Method for Feature Selection

Published Online: Jul 01, 2021

Page range: 10 - 28

Received: Aug 27, 2020

Accepted: Feb 22, 2021

DOI: https://doi.org/10.2478/cait-2021-0016

Keywords
Feature selection, data mining, random forest, Geni impurity, variable importance

© 2021 Yassine Akhiat et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

A New Noisy Random Forest Based Method for Feature Selection

Published Online: Jul 01, 2021

Page range: 10 - 28

Received: Aug 27, 2020

Accepted: Feb 22, 2021

DOI: https://doi.org/10.2478/cait-2021-0016

KeywordsFeature selection, data mining, random forest, Geni impurity, variable importance

© 2021 Yassine Akhiat et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
Feature selection, data mining, random forest, Geni impurity, variable importance