<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Aiming at the problems of low efficiency, low cost of time and space, this paper proposes an incremental association rule mining algorithm based on Hadoop load balancing. In combination with the tree structure, when the data in the database is constantly updated, it does not need to scan the original database to generate frequent item sets, and use the load balancing in the data distribution so that the master node distributes the data to the child nodes evenly. In the experiment of control variable method, the variables of minimum support and sample increment are processed respectively. The experimental results show that when the minimum support is unchanged and the transaction data set is increased, the incremental association rule mining algorithm based on Hadoop load balancing takes less than 14.3% of the Apriori algorithm. The number of association rules mined by the algorithm is more than that of the Apriori algorithm. And the memory usage of the Hadoop-based incremental association rule mining algorithm is much smaller than the Apriori algorithm; when the total amount of transaction data is constant and the minimum support is changed, the memory usage of the Hadoop-based incremental association rule mining algorithm is smaller than the Apriori algorithm. The Hadoop-based incremental association rule mining algorithm has some improvements in memory usage and efficiency.</p></abstract>

Aiming at the problems of low efficiency, low cost of time and space, this paper proposes an incremental association rule mining algorithm based on Hadoop load balancing. In combination with the tree structure, when the data in the database is constantly updated, it does not need to scan the original database to generate frequent item sets, and use the load balancing in the data distribution so that the master node distributes the data to the child nodes evenly. In the experiment of control variable method, the variables of minimum support and sample increment are processed respectively. The experimental results show that when the minimum support is unchanged and the transaction data set is increased, the incremental association rule mining algorithm based on Hadoop load balancing takes less than 14.3% of the Apriori algorithm. The number of association rules mined by the algorithm is more than that of the Apriori algorithm. And the memory usage of the Hadoop-based incremental association rule mining algorithm is much smaller than the Apriori algorithm; when the total amount of transaction data is constant and the minimum support is changed, the memory usage of the Hadoop-based incremental association rule mining algorithm is smaller than the Apriori algorithm. The Hadoop-based incremental association rule mining algorithm has some improvements in memory usage and efficiency.

Incremental Association Rule Mining Algorithm Based on Hadoop

International Journal of Advanced Network, Monitoring and Controls

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"Incremental Association Rule Mining Algorithm Based on Hadoop"}

Aiming at the problems of low efficiency, low cost of time and space, this paper proposes an incremental association rule mining algorithm based on Hadoop load...

symbol	means
DB	raw data set
db	new data set
DBUdb	all data sets
L_k	frequent item sets(k-order)
C_k	candidate set(k-order)

Item	TID
butter	1,5,6
citrus fruit	1,3
coffee	2,5
cream cheese	4,6
ham	1,5,6
newspapers	3,6
sauces	5,6
tropical fruit	2,4
whole milk	3,4
yogurt	1,2,4,5

Number of rules (n_k)	x_k	f(x_k	First order difference quotient	Second order difference quotient
1	0.0625	0.6
4	0.25	0.4	-1.07
7	0.4375	0.3	-0.53	1.44

TID	Item
1	citrus fruit, yogurt, butter, ham
2	tropical fruit, yogurt, coffee
3	whole milk, citrus fruit, newspapers
4	tropical fruit, yogurt, cream cheese, whole milk
5	coffee, butter, yogurt, sauces, ham
6	butter, ham, cream cheese, sauces, newspapers

CD\RD	ham	bread	coffee
butter	(0.4,0.55)	(0.13,0.28)	(0.2,0.3)
yogurt	(0.36,0.54)	(0.63,0.58)	(0.26,0.39)
cheese	(0.43,0.51)	(0.3,0.53)	(0.31,0.52)

Incremental Association Rule Mining Algorithm Based on Hadoop

Data publikacji: 14 paź 2019

Zakres stron: 7 - 16

DOI: https://doi.org/10.21307/ijanmc-2019-015

Słowa kluczowe
Hadoop, Tree Structure, Load Balancing, Frequent item Sets, Association Rules

© 2018 Zhu Ying et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Figure 7.

SYMBOLIC DESCRIPTION

VERTICAL DATABASE

DIFFERENCE QUOTIENT TABLE

HORIZONTAL DATABASE

RULE SUPPORT AND CONFIDENCE CALCULATION RESULTS

Incremental Association Rule Mining Algorithm Based on Hadoop

Data publikacji: 14 paź 2019

Zakres stron: 7 - 16

DOI: https://doi.org/10.21307/ijanmc-2019-015

Słowa kluczoweHadoop, Tree Structure, Load Balancing, Frequent item Sets, Association Rules

© 2018 Zhu Ying et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Figure 7.

SYMBOLIC DESCRIPTION

VERTICAL DATABASE

DIFFERENCE QUOTIENT TABLE

HORIZONTAL DATABASE

RULE SUPPORT AND CONFIDENCE CALCULATION RESULTS

Słowa kluczowe
Hadoop, Tree Structure, Load Balancing, Frequent item Sets, Association Rules