<abstract xmlns="http://www.w3.org/1999/xhtml"><p> When running data-mining algorithms on big data platforms, a parallel, distributed framework, such asMAPREDUCE, may be used. However, in a parallel framework, each individual model fits the data allocated to its own computing node without necessarily fitting the entire dataset. In order to induce a single consistent model, ensemble algorithms such as majority voting, aggregate the local models, rather than analyzing the entire dataset directly. Our goal is to develop an efficient algorithm for choosing one representative model from multiple, locally induced decision-tree models. The proposed SySM (syntactic similarity method) algorithm computes the similarity between the models produced by parallel nodes and chooses the model which is most similar to others as the best representative of the entire dataset. In 18.75% of 48 experiments on four big datasets, SySM accuracy is significantly higher than that of the ensemble; in about 43.75% of the experiments, SySM accuracy is significantly lower; in one case, the results are identical; and in the remaining 35.41% of cases the difference is not statistically significant. Compared with ensemble methods, the representative tree models selected by the proposed methodology are more compact and interpretable, their induction consumes less memory, and, as confirmed by the empirical results, they allow faster classification of new records.</p></abstract>

When running data-mining algorithms on big data platforms, a parallel, distributed framework, such asMAPREDUCE, may be used. However, in a parallel framework, each individual model fits the data allocated to its own computing node without necessarily fitting the entire dataset. In order to induce a single consistent model, ensemble algorithms such as majority voting, aggregate the local models, rather than analyzing the entire dataset directly. Our goal is to develop an efficient algorithm for choosing one representative model from multiple, locally induced decision-tree models. The proposed SySM (syntactic similarity method) algorithm computes the similarity between the models produced by parallel nodes and chooses the model which is most similar to others as the best representative of the entire dataset. In 18.75% of 48 experiments on four big datasets, SySM accuracy is significantly higher than that of the ensemble; in about 43.75% of the experiments, SySM accuracy is significantly lower; in one case, the results are identical; and in the remaining 35.41% of cases the difference is not statistically significant. Compared with ensemble methods, the representative tree models selected by the proposed methodology are more compact and interpretable, their induction consumes less memory, and, as confirmed by the empirical results, they allow faster classification of new records.

Interpretable decision-tree induction in a big data parallel framework

International Journal of Applied Mathematics and Computer Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Interpretable decision-tree induction in a big data parallel framework"}

When running data-mining algorithms on big data platforms, a parallel, distributed framework, such asMAPREDUCE, may be used. However, in a parallel framework,...

Interpretable decision-tree induction in a big data parallel framework

Data publikacji: 13 sty 2018

Zakres stron: 737 - 748

Otrzymano: 30 lis 2016

Przyjęty: 08 sie 2017

DOI: https://doi.org/10.1515/amcs-2017-0051

Słowa kluczowe
big data, parallel computing, MAPREDUCE, decision trees, editing distance, tree similarity

© by Mark Last

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Interpretable decision-tree induction in a big data parallel framework

Data publikacji: 13 sty 2018

Zakres stron: 737 - 748

Otrzymano: 30 lis 2016

Przyjęty: 08 sie 2017

DOI: https://doi.org/10.1515/amcs-2017-0051

Słowa kluczowebig data, parallel computing, MAPREDUCE, decision trees, editing distance, tree similarity

© by Mark Last

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Słowa kluczowe
big data, parallel computing, MAPREDUCE, decision trees, editing distance, tree similarity