<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis.201619_s_005_w2aab2b8c28b1b7b1aab1c15b1Aa"><h3>Purpose</h3><p>The authors aim at testing the performance of a set of machine learning algorithms that could improve the process of data cleaning when building datasets.</p></sec><sec id="j_jdis.201619_s_006_w2aab2b8c28b1b7b1aab1c15b2Aa"><h3>Design/methodology/approach</h3><p>The paper is centered on cleaning datasets gathered from publishers and online resources by the use of specific keywords. In this case, we analyzed data from the Web of Science. The accuracy of various forms of automatic classification was tested here in comparison with manual coding in order to determine their usefulness for data collection and cleaning. We assessed the performance of seven supervised classification algorithms (Support Vector Machine (SVM), Scaled Linear Discriminant Analysis, Lasso and elastic-net regularized generalized linear models, Maximum Entropy, Regression Tree, Boosting, and Random Forest) and analyzed two properties: accuracy and recall. We assessed not only each algorithm individually, but also their combinations through a voting scheme. We also tested the performance of these algorithms with different sizes of training data. When assessing the performance of different combinations, we used an indicator of coverage to account for the agreement and disagreement on classification between algorithms.</p></sec><sec id="j_jdis.201619_s_007_w2aab2b8c28b1b7b1aab1c15b3Aa"><h3>Findings</h3><p>We found that the performance of the algorithms used vary with the size of the sample for training. However, for the classification exercise in this paper the best performing algorithms were SVM and Boosting. The combination of these two algorithms achieved a high agreement on coverage and was highly accurate. This combination performs well with a small training dataset (10%), which may reduce the manual work needed for classification tasks.</p></sec><sec id="j_jdis.201619_s_008_w2aab2b8c28b1b7b1aab1c15b4Aa"><h3>Research limitations</h3><p>The dataset gathered has significantly more records related to the topic of interest compared to unrelated topics. This may affect the performance of some algorithms, especially in their identification of unrelated papers.</p></sec><sec id="j_jdis.201619_s_009_w2aab2b8c28b1b7b1aab1c15b5Aa"><h3>Practical implications</h3><p>Although the classification achieved by this means is not completely accurate, the amount of manual coding needed can be greatly reduced by using classification algorithms. This can be of great help when the dataset is big. With the help of accuracy, recall, and coverage measures, it is possible to have an estimation of the error involved in this classification, which could open the possibility of incorporating the use of these algorithms in software specifically designed for data cleaning and classification.</p></sec><sec id="j_jdis.201619_s_010_w2aab2b8c28b1b7b1aab1c15b6Aa"><h3>Originality/value</h3><p>We analyzed the performance of seven algorithms and whether combinations of these algorithms improve accuracy in data collection. Use of these algorithms could reduce time needed for manual data cleaning.</p></sec></abstract>

PurposeThe authors aim at testing the performance of a set of machine learning algorithms that could improve the process of data cleaning when building datasets.Design/methodology/approachThe paper is centered on cleaning datasets gathered from publishers and online resources by the use of specific keywords. In this case, we analyzed data from the Web of Science. The accuracy of various forms of automatic classification was tested here in comparison with manual coding in order to determine their usefulness for data collection and cleaning. We assessed the performance of seven supervised classification algorithms (Support Vector Machine (SVM), Scaled Linear Discriminant Analysis, Lasso and elastic-net regularized generalized linear models, Maximum Entropy, Regression Tree, Boosting, and Random Forest) and analyzed two properties: accuracy and recall. We assessed not only each algorithm individually, but also their combinations through a voting scheme. We also tested the performance of these algorithms with different sizes of training data. When assessing the performance of different combinations, we used an indicator of coverage to account for the agreement and disagreement on classification between algorithms.FindingsWe found that the performance of the algorithms used vary with the size of the sample for training. However, for the classification exercise in this paper the best performing algorithms were SVM and Boosting. The combination of these two algorithms achieved a high agreement on coverage and was highly accurate. This combination performs well with a small training dataset (10%), which may reduce the manual work needed for classification tasks.Research limitationsThe dataset gathered has significantly more records related to the topic of interest compared to unrelated topics. This may affect the performance of some algorithms, especially in their identification of unrelated papers.Practical implicationsAlthough the classification achieved by this means is not completely accurate, the amount of manual coding needed can be greatly reduced by using classification algorithms. This can be of great help when the dataset is big. With the help of accuracy, recall, and coverage measures, it is possible to have an estimation of the error involved in this classification, which could open the possibility of incorporating the use of these algorithms in software specifically designed for data cleaning and classification.Originality/valueWe analyzed the performance of seven algorithms and whether combinations of these algorithms improve accuracy in data collection. Use of these algorithms could reduce time needed for manual data cleaning.

Can Automatic Classification Help to Increase Accuracy in Data Collection?

United Kingdom of Great Britain and Northern Ireland

Science Policy Research Unit (SPRU), School of Business, Management and Economics

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Can Automatic Classification Help to Increase Accuracy in Data Collection?"}

PurposeThe authors aim at testing the performance of a set of machine learning algorithms that could improve the process of data cleaning when building...

Training set	10%	20%	50%	80%
Default model	73.56%	73.60%	73.11%	69.94%
Forest	80.69%	96.57%	96.52%	96.72%
GLMNet	82.88%	82.58%	94.98%	93.26%
Boosting	95.45%	95.22%	96.68%	95.18%
MaxEnt	86.95%	90.44%	92.89%	93.83%
SLDA	73.56%	74.90%	75.66%	88.25%
SVM	94.29%	95.80%	96.75%	98.07%
Tree	94.38%	95.56%	95.13%	93.83%

Training set	10%		20%		50%		80%

Algorithm	No	Yes	No	Yes	No	Yes	No	Yes
Forest	26.95%	100.00%	89.95%	98.95%	90.23%	98.84%	89.74%	99.72%
GLMNet	35.55%	99.88%	34.37%	99.87%	88.22%	97.46%	88.46%	95.32%
Boosting	93.34%	96.21%	95.43%	95.15%	96.26%	96.83%	90.38%	97.25%
MaxEnt	51.14%	99.82%	64.35%	99.80%	73.85%	99.89%	79.49%	100.00%
SLDA	0	100.00%	11.88%	97.51%	44.25%	87.21%	60.90%	100.00%
SVM	84.74%	97.72%	93.78%	96.52%	95.69%	97.15%	97.44%	98.35%
Tree	90.10%	95.92%	91.04%	97.18%	88.79%	97.46%	91.03%	95.04%

	Search string	Number of items
# 1	TS=“her 2”	8,542
# 2	String 1	26,972
# 3	#2 AND #1	6,396
# 4	#1 NOT #3	2,146

Consensus (Number of algorithms)	Coverage	Coverage No	Coverage Yes	Accuracy	Recall for the Yes category	Recall for the No category
≥ 4	2,330	616	1,714	87.51%	99.88%	53.08%
≥ 5	2,045	336	1,709	93.99%	99.94%	63.69%
≥ 6	1,825	162	1,663	98.19%	100.00%	79.63%
≥ 7	1,606	11	1,595	99.32%	100.00%	N/A

Can Automatic Classification Help to Increase Accuracy in Data Collection?

Article Category: Research Paper

Online veröffentlicht: 01. Sept. 2017

Seitenbereich: 42 - 58

Eingereicht: 17. Juli 2016

Akzeptiert: 29. Juli 2016

DOI: https://doi.org/10.20309/jdis.201619

Schlüsselwörter
Disambiguation, Machine learning, Data cleaning, Classification, Accuracy, Recall, Coverage

© 2016 Frederique Lang, Diego Chavarro, Yuxian Liu

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Accuracy – Individual benchmark.

Recall of individual algorithms.

Search process.

Performance of the two best algorithms combined.

Consensus on the classification of records.

Can Automatic Classification Help to Increase Accuracy in Data Collection?

Article Category: Research Paper

Online veröffentlicht: 01. Sept. 2017

Seitenbereich: 42 - 58

Eingereicht: 17. Juli 2016

Akzeptiert: 29. Juli 2016

DOI: https://doi.org/10.20309/jdis.201619

SchlüsselwörterDisambiguation, Machine learning, Data cleaning, Classification, Accuracy, Recall, Coverage

© 2016 Frederique Lang, Diego Chavarro, Yuxian Liu

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Accuracy – Individual benchmark.

Recall of individual algorithms.

Search process.

Performance of the two best algorithms combined.

Consensus on the classification of records.

Schlüsselwörter
Disambiguation, Machine learning, Data cleaning, Classification, Accuracy, Recall, Coverage