<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>We propose a Slovak language model for the spaCy library in Python. These models are easy-to-use for basic natural language processing tasks in a single package. The package contains several components for basic preprocessing tasks, such as tokenization, sentence boundary detection, syntactic parsing, lemmatization, named entity recognition, morphology analysis, and word vectors. It is based on the state-of-the-art monolingual SlovakBERT model. Named entity recognition is trained on a separate, publicly available WikiAnn database. The other statistical classifiers use a Slovak Dependency Treebank corpus. Morphological tags are compatible with the conventions of the Slovak National Corpus. The part of speech tags use conventions of the Universal Dependencies framework. We trained a separate word vector model on a web-based corpus. The training uses fastText with Floret modification. We present a series of experiments that confirm that the model performs similarly to other languages for all tasks. Training scripts and data are publicly available.</p>
</abstract>

We propose a Slovak language model for the spaCy library in Python. These models are easy-to-use for basic natural language processing tasks in a single package. The package contains several components for basic preprocessing tasks, such as tokenization, sentence boundary detection, syntactic parsing, lemmatization, named entity recognition, morphology analysis, and word vectors. It is based on the state-of-the-art monolingual SlovakBERT model. Named entity recognition is trained on a separate, publicly available WikiAnn database. The other statistical classifiers use a Slovak Dependency Treebank corpus. Morphological tags are compatible with the conventions of the Slovak National Corpus. The part of speech tags use conventions of the Universal Dependencies framework. We trained a separate word vector model on a web-based corpus. The training uses fastText with Floret modification. We present a series of experiments that confirm that the model performs similarly to other languages for all tasks. Training scripts and data are publicly available.

Slovak Language Models for Basic Preprocessing Tasks in Python

Journal of Linguistics/Jazykovedný casopis

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"Slovak Language Models for Basic Preprocessing Tasks in Python"}

We propose a Slovak language model for the spaCy library in Python. These models are easy-to-use for basic natural language processing tasks in a single...

Slovak Language Models for Basic Preprocessing Tasks in Python

Publié en ligne: 25 déc. 2023

Pages: 323 - 332

DOI: https://doi.org/10.2478/jazcas-2023-0049

Mots clés
Slovak language model, transformers, natural language processing

© 2023 Daniel Hládek et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Slovak Language Models for Basic Preprocessing Tasks in Python

Publié en ligne: 25 déc. 2023

Pages: 323 - 332

DOI: https://doi.org/10.2478/jazcas-2023-0049

Mots clésSlovak language model, transformers, natural language processing

© 2023 Daniel Hládek et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Mots clés
Slovak language model, transformers, natural language processing