A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Administered by WIPO, International Patent Classification (IPC) is the most essential classification system and the primary reference for patent classification. The areas of technology are split across eight sections (Table 1).

Each section is subdivided into classes, subclasses, groups, and subgroups, counting in the last level approximately 72,000 sub-groups (Espacenet, 2021).

In Portugal, Instituto Nacional de Propriedade Industrial (INPI) is the governmental agency responsible for protecting Industrial Property rights. Annually, INPI discloses statistical reports of Industrial Property in Portugal. After some decrease at the beginning of the decade, an increase of patents applications has been reported since 2016, with a growth of 40% from 2019 compared to the previous year (Figure 2).

Figure 2

Regarding the Portuguese patents, and following a general trend, the distribution among the categories is highly unbalanced and tends to follow a Pareto-like distribution, where about 80% of the documents fall into about 20% of the categories.

Text classification has been one of the most known text mining downstream tasks. Intelligent text classification methods provide superior facilities, save time and money while handling the increase of digital texts we are facing (Manning, Raghavan, & Schutze, 2008; Silva & Ribeiro, 2010). The success of text mining is highly dependent on the preprocessing phase that deals with the transformations of text into structured data sets. The challenge here is how to represent the complex characteristics of word uses and their variations across linguistics contexts in a structured way. It is not a trivial task since language by itself is complex, and its production and comprehension traverse many levels of linguistic analysis, like morphological, lexical, syntactical, semantical, and pragmatical (Feldman & Sanger, 2006; Liddy, 2001; Peters et al., 2018). Nevertheless, the lack of labelled training instances or the cost to train a model with a massive amount of data can be a limitation for specific real-world applications. To overcome these challenges, the strategy of transferring the knowledge across domains instead of training from scratch, which had a large impact on computer vision, has been revolutionising text mining. The transfer learning approach and pre-trained language models have been achieving the state-of-art in different tasks (Pan & Yang, 2010; Zhuang et al., 2021).

Several methods were proposed for addressing the patent classification task and a survey on automatic patent classification systems and traditional machine learning for text classification and clustering were proposed by Gomez and Moens (2014). Their analysis was done with a complete explanation of patents and the classification system. Furthermore, they compared studies with several algorithms, distinct feature processing methods and different levels of classification in IPC. They conclude that patent classification is a complex problem which there are still many alternatives to explore.

After defining “Intellectual Property Analytics (IPA) as the data science of analysing a large amount of Intellectual Property information to discover relationships, trends and patterns for decision”, Aristodemou and Tietze (2018) reviewed 57 articles. They promoted a discussion about AI, machine learning and deep learning approaches to analyse IPA data. They observed the growth of publications over the years, with most of them concentrated in computer science subjects. Moreover, they call attention to the high concentration of articles around artificial neural networks (ANN) and the use of backpropagation learning methods, followed by support vector machine (SVM) and conditional random fields (CRF), focused on classification tasks (Aristodemou & Tietze, 2018).

Zhang (2014) used SVM to build sub-classifiers for each class of the patents that are combined in a multi-classifier fusion where the final label is selected using an active learning method Wu et al. (2016) proposed a patent classification system based on a self-organising map (SOM), kernel principal component analysis (KPCA) and SVM, using quality indicators extracted from different parts of the document. Trappey et al. (2006) presented a patent document classification method based on neural network and critical phrases frequency that claimed to yield average accuracy above 90% in a specific test set. Further, the same authors used an ontology-based artificial neural network to automatically classify and search knowledge documents (Trappey et al., 2013).

DeepPatent is a deep learning algorithm proposed by Li et al. (2018) that combines word embedding pre-trained on the title and abstract sections using skip-gram and a Convolutional Neural Network (CNN) with multi-size filters. This approach aimed to overcome the main limitations of the traditional text encoding and machine learning algorithms, like data sparsity, the inability of capturing complex contents and poor performance on large datasets. Their model presented an F1 top 4 score of 55.09%. A recent study compares the performance of a hierarchical SVM and various neural network models. It applies state-of-the-art hyperparameter optimisation techniques on a CNN to understand the effects on the model accuracy. With this optimised neural network, they achieved 55.02% accuracy on the public Wipo-Alpha dataset

https://www.wipo.int/classifications/ipc/en/ITsupport/Categorization/dataset/index.html

Derieux et al. (2010) used different approaches to classify CLEF-IP dataset with English (68%), German (24%) and French (8%) patents. They found different results according to the language. Indeed, classification on German patents was not less than 10 points below English patent classification. Albeit the size of the training set has a significant impact on these results, they impute this difference to the specificities of each language as well.

Concerned about the specificity of the language used in patent applications, Risch and Krestel (2019) proposed domain-specific word embeddings. They trained a fastText model on a dataset of more than 5 million patents in English and evaluated it at the WIPO-alpha dataset through a bi-directional Gated Recurrent Units (GRUs). They concluded that domain-specific representations outperform those trained on Wikipedia, but the underrepresented classes still were a challenge in this problem.

Following the Australasian Language Technology Association Workshop, with the challenge of fine-tuning a pre-trained language model to classify Australian patents, Lee and Hsiang (2020) pre-trained BERT in a dataset of approximately 3 million documents from Google Patents Public Datasets, using the claims section only and compared their results with the DeepPatent ones. They achieve an F1 score of 63.74% in IPC subclass level (632 labels). Table 3 summarises the studies mentioned above.

More recently a survey focused on deep learning approaches for different patent analysis tasks was proposed (Krestel et al., 2021). The authors covered different tasks such as patent classification, technology forecasting (where patents are used to assess a technology landscape), patent text generation (where the structure and styles incorporated in published patent documents are used to automate the process of writing patent claims), and computer vision problems (considering figures and drawings from patent documents instead of text).

In particular, in their review the authors discussed advantages and disadvantages of different deep learning approaches for the considered tasks. Deep learning classification models have been widely used for automatic patent classification tasks for taking advantage of the more complex architectures. However, other approaches can also be considered, such as training representations to improve clustering, and classifying citations into applicant-provided or examiner-provided (Krestel et al., 2021). For this one, methods like PCA (principal components analysis) can be used to classify the text according to the co-citation relationship (Lay & Wu, 2005). The point of attention here is for the link between two patents. When it is negative, it can point to an inaccurate relationship (Trappey et al., 2006). In addition, the practice of citations differs between patent offices and may not be fully available in the text.

Regardless of not having found any study about patent classification in Portuguese, there are several studies about Portuguese text classification using machine learning algorithms (Gonçalves et al., 2006). In the same way, pre-trained word-embeddings have been used in other text mining use cases (De Castro et al., 2019; Rodrigues et al., 2020). To perform a Named entity recognition (NER) task, dos Santos and Guimarães (2015) pre-trained word-level embeddings using word2Vec in a corpus composed by Portuguese Wikipedia, CETENFolha

https://www.linguateca.pt/cetenfolha/

Two types of .xml files with general information of granted patents manually classified were made available to this study. The first one with bibliographic data (patent id, applicants, inventors), title, abstract, IPC codes and CPC codes. The second one with patent id, description and claims. After joining the data, it was counted approximately 40,000 instances and 124 different categories considering the IPC second level and only the main classification. The first step was to exclude those instances that have no abstract, claims and description data. Secondly, duplicated patents were identified. It was considered duplicated instances ones with identical id. In these cases, only the first instance was kept to preserve the main IPC classification code. Then, the IPC code was split into 5 columns according to its levels. Finally, a new feature was created by the concatenation of the title and claims or abstract features (Table 4).

Many related studies have used title and abstract as input to the classification model. However, more than 80% of the instances in this dataset had no value for abstract while claims and description features had about 0.3% of missing values and all the instances had title information. Therefore, claims were chosen to be used as the input to the classification model. Further, it brings important but more concise information about the patent application than description. For those instances without claims, abstract data was used.

The final dataset had 36,100 instances.

Feature engineering is a critical step in text mining tasks. The goal is to choose the best way to represent the words of the text to be able to apply the algorithms (Kowsari et al., 2019). Different transformations were applied according to the experiment path. The first approach was to apply TF-IDF to represent the features. To do so, we started cleaning the text and deleting stopwords. Then the cleaned text was tokenised and vectorised. Even knowing that specific domain-language embeddings bring better performance (Risch & Krestel, 2019), as our dataset was not big enough to train a reasonable Language Model, we decided to validate pre-trained Portuguese word embeddings. We explored word vectors that have been trained in Portuguese corpus, using Word2Vec, Glove, and FastText architectures, on both CBOW and skip-gram strategies. Most of them were well succeeded in recognising the vocabulary used in the patent dataset. But FastText models can represent words that are not in the original dataset. For the others, in general, the worst performance occurred in section C and this can be explained by the section topic (Chemistry and Metallurgy) and the technical words used to describe these kinds of inventions.

So, as a second approach, we again cleaned and tokenised the text and applied FastText word embedding pre-trained on Common Crawl and Wikipedia using CBOW with position-weights, in dimension 300, with character n-grams of length five and window of size 5. For the pre-trained models, the feature engineering step must be performed using the methods available with the models and the dataset is transformed into the specific format that the model expected. In this case, the data was tokenised and the sentences prepared with the addition of special tokens. All the documents were truncated to the maximum length of 128 tokens.

Following the paths described previously, traditional machine learning algorithms and ensemble methods were tested, namely Logistic Regression, Linear Support Vector Machine (SVM), Linear model with Stochastic Gradient Descent (SGD), RandomForest, and XGBoost. Starting with linear classifiers, Logistic Regression is one of the earliest and well-known classification algorithms and the SVM classifier method for many years has outstanding with its effectiveness in text classification. SGD learning allows minibatch and can be a good strategy for large-scale problems. Tree-based classification algorithms, especially voting classifiers like XGBoost, can be fast and accurate for document classification (Kowsari et al., 2019). To start training, after preparing the features by applying TF-IDF, the default algorithms parameters were applied, and the models were evaluated by cross-validation in 5 folds. Following, document embeddings were computed using two different strategies. Firstly, by the mean of the FastText word-embeddings. Secondly, by Doc2Vec, using the whole corpus. The same algorithms were run using the document vectors, and again, a cross-validation strategy was used. The best result was achieved with a LinearSVC model using TF-IDF. Then, a grid search was applied to it as a tuning strategy. In the second experiment path, two Neural Networks architectures were trained and optimised. A Convolutional Neural Network (CNN) and a bi-directional Long Short-Term Memory (BiLSTM). In both cases, FastText word-embeddings were applied. CNN can capture local correlations of spatial or temporal structures. In NLP tasks, it means extract n-gram features at different positions of text through a series of convolutional filters. These kinds of models have been achieving good performance in text classifications and also for some cases of patent classification (Hu et al., 2018; Li et al., 2018). LSTM was designed to handle sequence data and capture long-term dependence while controlling the ratio of information to forget and to store during the training. A BiLSTM is a combination of two LSTM in which the context is seen in both directions, from left to right (forward) and from right to left (backward). It means, for each word, capture previous and following information, with the ability to remember or forget it when necessary. In the end, the weights of the two networks are combined to compute the output. For this property of handle long-term dependencies, BiLSTM has been widely used in text classification (Bispo et al., 2019; Devlin et al., 2019; Hu et al., 2018).

In the last experiment path, pre-trained BERT, DistilBERT, and ULMFiT models were applied. We started with BERT-Base Multilingual Cased, a model made available by BERT authors which supports 104 languages. Next, BERTimbau which is a BERT model trained on the BrWaC (Wagner Filho et al., 2019), a large Portuguese corpus, for 1,000,000 steps, using the wholeword mask. Then, DistilBERT Base Multilingual Cased, this model is based on BERT architecture, supports the same 104 languages, and uses the BERT tokeniser, but it is built with only 6 layers, half of the BERT-Base model. It aims to be lighter and to run faster, since it has fewer parameters, while preserving a good performance. For BERT models several warmup values, text length, and hyperparameters were tried. Finally, a hyperparameter tuning and unfreeze strategy were applied to ULMFiT.

The dataset was high imbalanced, with the two most populated counting about 21% and 15% of the dataset, respectively, and the least populated ones having a single representative. Therefore, we randomly selected 2,000 patents for the two most populated classes, excluded those with just one patent, and kept the remaining untouched to create the final dataset. To handle the imbalance, the models’ hyperparameter “classweight” was set to “balanced”. This parameter is used during the training phase to penalize the misclassification made by the minority classes and, at the same time, to reduce the weight of the majority classes’ errors. For the models that did not have this option, we considered the metric F1 weighted that computes the average weighted by the number of true instances for each class.

To check the performance of the models, the dataset was split into a training set with 25,267 patents and the test set with the remaining 30% of the documents. For each experiment path, after training the models, the unseen test set was prepared with the required transformations and submitted to the model to predict the patent class. We used the predicted and real labels to compute Precision, Recall, F1 and F1 weighted average as evaluation metrics.

Precision is the number of patents correctly classified as a percentage of all the patents classified in that class Precision=TPTP+FP. Precision = {{TP} \over {TP + FP}}.

Recall is the number of patents correctly classified as a percentage of all the patents belonging to that class Recall=TPTP+FN. Recall = {{TP} \over {TP + FN}}.

F1, also known as F-score or F, is the harmonic mean of precision and TP rate F1=2×Precision×RecallPrecision+Recall. F1 = {{2 \times Precision \times {{Recall}}} \over {Precision + {{Recall}}}}.

F1 weighted average is the average of F1 for each class weighted by the number of items of each label in the actual data.