<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or irrelevant to a particular literature study topic. Conventional machine learning algorithms for tabular data which can learn quickly from not necessarily large and usually imbalanced data with low computational demands are well suited to this application, but they require that the text data be transformed to a vector representation. This work investigates the utility of different types of text representations for this purpose. Experiments are presented using the bag of words representation and selected representations based on word or text embeddings: <italic>word2vec, doc2vec, GloVe, fastText, Flair</italic>, and <italic>BioBERT</italic>. Four classification algorithms are used with these representations: a naive Bayes classifier, logistic regression, support vector machines, and random forest. They are applied to datasets consisting of scientific article abstracts from systematic literature review studies in the medical domain and compared with the pre-trained <italic>BioBERT</italic> model fine-tuned for classification. The obtained results confirm that the choice of text representation is essential for successful text classification. It turns out that, while the standard bag of words representation is hard to beat, <italic>fastText</italic> word embeddings make it possible to achieve roughly the same level of classification quality with the added benefit of much lower dimensionality and capability of handling out-of-vocabulary words. More refined embeddings methods based on deep neural networks, while much more demanding computationally, do not appear to offer substantial advantages for the classification task. The fine-tuned <italic>BioBERT</italic> classification model performs on par with conventional algorithms when they are coupled with their best text representation methods.</p>
</abstract>

Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or irrelevant to a particular literature study topic. Conventional machine learning algorithms for tabular data which can learn quickly from not necessarily large and usually imbalanced data with low computational demands are well suited to this application, but they require that the text data be transformed to a vector representation. This work investigates the utility of different types of text representations for this purpose. Experiments are presented using the bag of words representation and selected representations based on word or text embeddings: word2vec, doc2vec, GloVe, fastText, Flair, and BioBERT. Four classification algorithms are used with these representations: a naive Bayes classifier, logistic regression, support vector machines, and random forest. They are applied to datasets consisting of scientific article abstracts from systematic literature review studies in the medical domain and compared with the pre-trained BioBERT model fine-tuned for classification. The obtained results confirm that the choice of text representation is essential for successful text classification. It turns out that, while the standard bag of words representation is hard to beat, fastText word embeddings make it possible to achieve roughly the same level of classification quality with the added benefit of much lower dimensionality and capability of handling out-of-vocabulary words. More refined embeddings methods based on deep neural networks, while much more demanding computationally, do not appear to offer substantial advantages for the classification task. The fine-tuned BioBERT classification model performs on par with conventional algorithms when they are coupled with their best text representation methods.

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

International Journal of Applied Mathematics and Computer Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or...

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

Published Online: Dec 21, 2023

Page range: 603 - 621

Received: Apr 19, 2023

Accepted: Aug 28, 2023

DOI: https://doi.org/10.34768/amcs-2023-0043

Keywords
text representation, text classification, bag of words, word embeddings

© 2023 Paweł Cichosz, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

Published Online: Dec 21, 2023

Page range: 603 - 621

Received: Apr 19, 2023

Accepted: Aug 28, 2023

DOI: https://doi.org/10.34768/amcs-2023-0043

Keywordstext representation, text classification, bag of words, word embeddings

© 2023 Paweł Cichosz, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
text representation, text classification, bag of words, word embeddings