<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Purpose</h3>
<p>Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has been increasing worldwide. We propose using machine learning algorithms to classify Portuguese patents and evaluate the performance of transfer learning methodologies to solve this task.</p>
</sec>
<sec><h3>Design/methodology/approach</h3>
<p>We applied three different approaches in this paper. First, we used a dataset available by INPI to explore traditional machine learning algorithms and ensemble methods. After preprocessing data by applying TF-IDF, FastText and Doc2Vec, the models were evaluated by cross-validation in 5 folds. In a second approach, we used two different Neural Networks architectures, a Convolutional Neural Network (CNN) and a bi-directional Long Short-Term Memory (BiLSTM). Finally, we used pre-trained BERT, DistilBERT, and ULMFiT models in the third approach.</p>
</sec>
<sec><h3>Findings</h3>
<p>BERTTimbau, a BERT architecture model pre-trained on a large Portuguese corpus, presented the best results for the task, even though with a performance of only 4% superior to a LinearSVC model using TF-IDF feature engineering.</p>
</sec>
<sec><h3>Research limitations</h3>
<p>The dataset was highly imbalanced, as usual in patent applications, so the classes with the lowest samples were expected to present the worst performance. That result happened in some cases, especially in classes with less than 60 training samples.</p>
</sec>
<sec><h3>Practical implications</h3>
<p>Patent classification is challenging because of the hierarchical classification system, the context overlap, and the underrepresentation of the classes. However, the final model presented an acceptable performance given the size of the dataset and the task complexity. This model can support the decision and improve the time by proposing a category in the second level of ICP, which is one of the critical phases of the grant patent process.</p>
</sec>
<sec><h3>Originality/value</h3>
<p>To our knowledge, the proposed models were never implemented for Portuguese patent classification.</p>
</sec>
</abstract>

Purpose
Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has been increasing worldwide. We propose using machine learning algorithms to classify Portuguese patents and evaluate the performance of transfer learning methodologies to solve this task.

Design/methodology/approach
We applied three different approaches in this paper. First, we used a dataset available by INPI to explore traditional machine learning algorithms and ensemble methods. After preprocessing data by applying TF-IDF, FastText and Doc2Vec, the models were evaluated by cross-validation in 5 folds. In a second approach, we used two different Neural Networks architectures, a Convolutional Neural Network (CNN) and a bi-directional Long Short-Term Memory (BiLSTM). Finally, we used pre-trained BERT, DistilBERT, and ULMFiT models in the third approach.

Findings
BERTTimbau, a BERT architecture model pre-trained on a large Portuguese corpus, presented the best results for the task, even though with a performance of only 4% superior to a LinearSVC model using TF-IDF feature engineering.

Research limitations
The dataset was highly imbalanced, as usual in patent applications, so the classes with the lowest samples were expected to present the worst performance. That result happened in some cases, especially in classes with less than 60 training samples.

Practical implications
Patent classification is challenging because of the hierarchical classification system, the context overlap, and the underrepresentation of the classes. However, the final model presented an acceptable performance given the size of the dataset and the task complexity. This model can support the decision and improve the time by proposing a category in the second level of ICP, which is one of the critical phases of the grant patent process.

Originality/value
To our knowledge, the proposed models were never implemented for Portuguese patent classification.

A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution 4.0 International License.

{"article-title":"A Use Case of Patent Classification Using Deep Learning with Transfer Learning"}

Purpose
Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has...

A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Article Category: Research Paper

Publicado en línea: 12 ago 2022

Páginas: 49 - 70

Recibido: 12 mar 2022

Aceptado: 04 jul 2022

DOI: https://doi.org/10.2478/jdis-2022-0015

Palabras clave
Natural Language Processing (NLP), Patent classification, Transfer Learning, Bi-directional Encoder Representations for Transformers (BERT)

© 2022 Roberto Henriques et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Article Category: Research Paper

Publicado en línea: 12 ago 2022

Páginas: 49 - 70

Recibido: 12 mar 2022

Aceptado: 04 jul 2022

DOI: https://doi.org/10.2478/jdis-2022-0015

Palabras claveNatural Language Processing (NLP), Patent classification, Transfer Learning, Bi-directional Encoder Representations for Transformers (BERT)

© 2022 Roberto Henriques et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Palabras clave
Natural Language Processing (NLP), Patent classification, Transfer Learning, Bi-directional Encoder Representations for Transformers (BERT)