<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or irrelevant to a particular literature study topic. Conventional machine learning algorithms for tabular data which can learn quickly from not necessarily large and usually imbalanced data with low computational demands are well suited to this application, but they require that the text data be transformed to a vector representation. This work investigates the utility of different types of text representations for this purpose. Experiments are presented using the bag of words representation and selected representations based on word or text embeddings: <italic>word2vec, doc2vec, GloVe, fastText, Flair</italic>, and <italic>BioBERT</italic>. Four classification algorithms are used with these representations: a naive Bayes classifier, logistic regression, support vector machines, and random forest. They are applied to datasets consisting of scientific article abstracts from systematic literature review studies in the medical domain and compared with the pre-trained <italic>BioBERT</italic> model fine-tuned for classification. The obtained results confirm that the choice of text representation is essential for successful text classification. It turns out that, while the standard bag of words representation is hard to beat, <italic>fastText</italic> word embeddings make it possible to achieve roughly the same level of classification quality with the added benefit of much lower dimensionality and capability of handling out-of-vocabulary words. More refined embeddings methods based on deep neural networks, while much more demanding computationally, do not appear to offer substantial advantages for the classification task. The fine-tuned <italic>BioBERT</italic> classification model performs on par with conventional algorithms when they are coupled with their best text representation methods.</p>
</abstract>

Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or irrelevant to a particular literature study topic. Conventional machine learning algorithms for tabular data which can learn quickly from not necessarily large and usually imbalanced data with low computational demands are well suited to this application, but they require that the text data be transformed to a vector representation. This work investigates the utility of different types of text representations for this purpose. Experiments are presented using the bag of words representation and selected representations based on word or text embeddings: word2vec, doc2vec, GloVe, fastText, Flair, and BioBERT. Four classification algorithms are used with these representations: a naive Bayes classifier, logistic regression, support vector machines, and random forest. They are applied to datasets consisting of scientific article abstracts from systematic literature review studies in the medical domain and compared with the pre-trained BioBERT model fine-tuned for classification. The obtained results confirm that the choice of text representation is essential for successful text classification. It turns out that, while the standard bag of words representation is hard to beat, fastText word embeddings make it possible to achieve roughly the same level of classification quality with the added benefit of much lower dimensionality and capability of handling out-of-vocabulary words. More refined embeddings methods based on deep neural networks, while much more demanding computationally, do not appear to offer substantial advantages for the classification task. The fine-tuned BioBERT classification model performs on par with conventional algorithms when they are coupled with their best text representation methods.

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

International Journal of Applied Mathematics and Computer Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Text classification has become a standard component of automated systematic literature review (SLR) solutions, where articles are classified as relevant or...

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

Data publikacji: 21 gru 2023

Zakres stron: 603 - 621

Otrzymano: 19 kwi 2023

Przyjęty: 28 sie 2023

DOI: https://doi.org/10.34768/amcs-2023-0043

Słowa kluczowe
text representation, text classification, bag of words, word embeddings

© 2023 Paweł Cichosz, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Bag of Words and Embedding Text Representation Methods for Medical Article Classification

Data publikacji: 21 gru 2023

Zakres stron: 603 - 621

Otrzymano: 19 kwi 2023

Przyjęty: 28 sie 2023

DOI: https://doi.org/10.34768/amcs-2023-0043

Słowa kluczowetext representation, text classification, bag of words, word embeddings

© 2023 Paweł Cichosz, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Słowa kluczowe
text representation, text classification, bag of words, word embeddings