Identifying multidisciplinary problems from scientific publications based on a text generation method

The module employs a text classification technique to assign a research objective label and a disciplinary code for each paper. The research objective classification model and the disciplinary classification model were created to implement the annotation of research objective types and disciplinary codes.

The abstract text of a paper is usually regarded as a summary description of the paper, whereas the title is a more condensed summary description than the abstract. Therefore, generating titles based on abstract texts will be effective. There are many effective technical methods for generating abstractive text, taking the content of an article as input based on pre-trained language models, such as Bert, T5, and Bart. These techniques generate a short and concise summary that captures the salient ideas of the source text and can be employed to generate abstractive titles. A recent study used text generation technology to generate special formatted titles using abstracts known as “A based on B” (Cheng et al., 2021). From some examples provided in the study, the method could produce good results when generating text, but there were still some poor results. The reason for the poor results is likely that the study considered only one type of research objective. Owing to significant differences in the abstractive titles corresponding to the three types of research objectives (US, SO, and EXP), specialized text generation models are constructed for each type to ensure optimal output. During the process of generation, the corresponding generation models are selected based on the labels annotated by the research objective classification model.

When creating abstractive title text generation models, the pre-trained models selected must be fine-tuned to achieve good results. During the fine-tuning process, some parameters were transferred from the pre-trained models, and external paper data were used to retrain the models. The abstracts in the external data are fed into pre-trained models, and titles in the external data are expected to be generated. To obtain external data, research paper data are first retrieved from a database that matches the format of the title for each type of research objective (see Table 1), and then the research objective classification model is used to assign labels to all papers. Finally, papers that match the title pattern in Table 1 are selected to form the training and testing datasets. A certain number of papers are required for each research object type in the external data. For example, in the experimental dataset, there were 11,000 external data points for each type of research objective, of which 10,000 were used for model training and 1,000 were used for model testing.

The abstractive titles generated by the text generation model may contain irrelevant content unrelated to the research problem. For example, in the generated abstractive title “Research on Low-Carbon Coal Blending Optimization of Thermal Power Units Based on Chaotic Particle Swarm Optimization”, “low-carbon coal blending optimization of thermal power units” is the phrase to be extracted as the research problem, which is only a part of the title. The remaining parts may include modifiers, research methods, or other words. Using the abstractive title generation models trained in the previous step, text with title patterns in Table 1 will be generated after taking the abstract as input. The purpose of this study is to extract problems; therefore, it is necessary to extract problem phrases solely based on the features analyzed in the titles.

The regular expression (regex) method was used to extract problem phrases. A regular expression is a special text string for describing a search pattern, mainly for use in pattern matching with strings or string matching. By analyzing the title features in Table 1, it can easily be seen that the position of problem phrases in the abstractive titles of each type of research objective is relatively fixed. For example, in the title of an US type paper “Research/Analysis of the Performance/Characteristics of A”, the content following words like “performance” or “characteristics” serves as the clues of occurring problems. Based on the results of these feature analyses, it is easy to compose regex rules to extract problem phrases from the generated abstractive titles.

The different forms of problem phrases extracted from the abstractive titles are likely to represent the same problems; therefore, using the same string as a criterion to determine the same problem will output a large number of incorrect problems. Therefore, it is essential to determine the same problem. Through an analysis of problem phrases, we found that these phrases typically contain nouns, verbs, prepositions, conjunctions, and other elements. However, nouns and verbs are often core elements that truly reflect the research problem. For example, in the problem phrase “preparation and catalytic oxidation of porous nanocatalysts for carbon monoxide,” the elements that truly reflect the research problem are the noun “preparation,” “oxidation,” “porous,” “nano,” “oxidation,” and “carbon monoxide.” These elements are called problem terms in this study. The combination of these terms can concisely describe the research problem in a paper. Generally, if the two problem phrases have the same terms, even if their textual descriptions are slightly different, they are likely the same problem. In another case, the two problem phrases share some common terms; although they are not exactly the same, they may be the same problem or different aspects of a problem, which is common in multidisciplinary research problems. Therefore, a process with three stages (Figure 2) is proposed to determine and merge the same problems. The first stage is to determine which terms can become problem terms, the second stage is to construct problem relation network based on common terms relationships between problems phrases, and the third stage is to identify same problems by using community detection algorithm.

Figure 2.

After merging the problem phrases into problems, non-duplicated research problems were obtained. Integrating the disciplinary classification codes of papers corresponding to problem phrases in the problem community can provide information for determining whether a problem involves multiple disciplines. For example, “catalysis, porous, oxidation” is a research problem that results from merging the research problems “Preparation of Porous Nano Catalysts and Catalytic Oxidation of Carbon Monoxide” from “0703 Chemistry” and “Preparation of Nitrogen-Doped Graphene Porous Carbon Composite Materials and Their Redox Catalysis” from “0805 Materials Science and Engineering.” As it involves two disciplinary fields, it is identified as a multidisciplinary research problem.

In the construction of the research objective classification model, five classical algorithms were employed to find the best model for annotating papers. The comparative experimental results are shown in Table 3. Precision, recall, and F1 score were used as evaluation metrics, and the overall performance of the classification model was assessed using macro-average metrics. It can be observed that the SVM algorithm achieved the best performance across all evaluation metrics. Therefore, the research objective classification model based on the SVM algorithm was selected for this labeling task. Using the trained model, all papers in the CPCN dataset were assigned one of four labels: US, SO, EXP-S, and EXP-RG.

In the experiment, because the metadata of the literature database did not provide disciplinary codes, it was necessary to establish a disciplinary classification model. As described in Section 3.2.1, a hierarchical text classification method based on stacking ensemble learning (Ran et al., 2020) was selected to assign a label of disciplinary code to each paper in the CPCN dataset. Similar to the construction of the research objective classification model, precision and recall were used to evaluate the three other algorithms. The goal was to determine whether the stacking algorithm was the most effective. Additionally, macro-precision, macro-recall, and macro-F1 score were selected as the evaluation metrics. The experimental results are listed in Table 4. The stacking method is optimal for all metrics, with F1 values increased by 0.06, 0.12, and 0.14, respectively, compared to the SVM, NB, and LSTM methods. Therefore, the disciplinary classification model based on the stacking algorithm was selected for this labeling task. Using the trained model, each paper in the CPCN dataset was assigned a first-level disciplinary code.

The pre-trained BART model (Lewis et al., 2020) and ChatGLM model (Du et al., 2022; Zeng et al., 2022) were used to create a text generation model for generating abstractive titles for papers. BART, a denoising autoencoder for pretraining sequence-to-sequence models, learns to map corrupted documents to the original documents and achieves state-of-the-art results on a number of text generation tasks. The ChatGLM model is based on the General Language Model (GLM) architecture with an autoregressive blank infilling method, specifically designed for generating natural language dialogues. Through unsupervised pre-training on a large-scale text corpus, the two models demonstrated good performance and versatility for various downstream tasks. Therefore, they can be used as a basis for fine-tuning a target task. Fine-tuning allows us to adapt the model to the target domain and task. As described in Section 3.2.2, 11,000 papers were retrieved as external data from the literature database according to the title pattern for each research objective in Table 1, of which 10,000 were used as fine-tuning data and 1,000 were used as testing data. The size of the data may have different effects on the results of fine-tuning the models, but this study focuses on the introduction of the presented method and did not compare the effects of fine-tuning the two models with different data volumes. These external finetuning data were fed into pre-trained models with abstracts as the input and titles as the expected generated text output. After all fine-tuning data were fed, the trained models generated an abstractive title with a specific text pattern when inputting the abstract of the paper.

To evaluate the performance of BART and ChatGLM in generating abstractive titles, the BLEU metric (Papineni et al. 2002) was chosen as the criterion. In machine translation technology, BLEU is a measurement of the difference between automatic translations and human-created reference translations of the same source sentence. Therefore, BLEU can also be used to determine the effectiveness of the generated text if the generated abstractive title is considered an automatic translation and the original title as the reference translation. The BLEU score was calculated by comparing the n-grams of the machine-translated sentences to the n-gram of the reference sentences. In general, 1-Gram is used to assess fidelity to the original text, while 2-Gram and 3-Gram reflect the fluency and readability of the sentence, respectively. To better evaluate the effectiveness of problem phrases, the unigram metric is employed to measure the matching degree between the generated problem phrase and the actual problem phrase at the character level, while the Exact Match metric is employed to evaluate the model’s ability to restore problem phrases from the original title at the vocabulary level by calculating the proportion of generated problem phrases that match exactly with those in the original problem phrases.

Table 5 shows the comparative results between BART and ChatGLM, where ALL in the first column means merging four types of external data as fine-tuning data to train one model, rather than separating them according to the research objectives and training the four models. First, we compared the text generation performance of the two models in every research objective model. It can be observed that BART outperformed ChatGLM in almost all metrics. ChatGLM is only slightly better than BART in the US model (Exact Match and Unigram) and the EXP-RG model (Unigram). This may be due to the fact that the abstractive titles generated by the ChatGLM model are longer and contain more characters than the original titles. Therefore, we trained only the ALL-BART model, which combines four types of external data. Compared to the four independent research objective models, most metrics of the ALL-BART model exhibited poor performance. Only the Exact Match metric ALL-BART model performed slightly better than the US-BART and EXP-RG-BART models did. This may be because when merging four types of external data, US and EXP-RG papers that perform poorly on this metric will be diluted by other types of papers, making the overall performance better than calculating US or EXP-RG type papers alone.

A current similar method based on text generation did not distinguish the research objectives of the study (Cheng et al., 2021). Although the study demonstrated through manual evaluation that the proposed method can generate high-quality titles describing problems and methods, the provided Turing test examples reveal that the generation model effect is not ideal and needs improvement. For example, the original title of a sample paper is “Network Intrusion Detection Based on the Fusion of Mean Clustering Analysis and Multilayer Core Set Aggregation Algorithm,” while the generated title is “Network Intrusion Detection Based on K-Means Algorithm.” Unlike existing methods, the method proposed in this article classifies papers based on research objectives and constructs text generation models separately. The All-Bart model in Table 5 did not distinguish research objectives like the current method, and its indicators were significantly lower than the models that distinguish research objectives, fully demonstrating the necessity of constructing a text-generation model to identify problems by distinguishing research objectives in papers.

To validate the effectiveness of the proposed method in identifying multidisciplinary research problems, experiments were conducted on the test set of the CPCN dataset. A total of 264 multidisciplinary research problems were identified in 3,879 papers. Table 6 presents a partial list of recognized multidisciplinary research problems and the first-level disciplines involved.

Fifty identified multidisciplinary research problems were randomly sampled and evaluated by three experts with backgrounds in materials science, energy and power, and information science. For each problem, all problem phrases and corresponding papers were used as evaluation materials, and only the unanimous agreement of three experts was considered an evaluation outcome. That is, only multidisciplinary problems identified by three experts simultaneously were considered as preliminary evaluation results. If they have different opinions on a certain problem, a search for dissertations containing all three terms of the problem will be conducted in the dissertation database of CNKI. The obtained theses were used to evaluate the multidisciplinary nature of the problem. The basic idea is that the metadata of theses contain information about the authors’ colleges or schools, and the division of colleges in universities is generally based on disciplines; therefore, if a problem is studied in multiple colleges or schools, it is likely to be multidisciplinary. Several studies have found that identifying multidisciplinary papers based on co-authors from different colleges or schools performed well (Li & Yu, 2018; Wang, et al., 2015), which provides support to the idea. The retrieved theses will then be ranked and sorted based on relevance, and the discipline or school department information from the top five theses will be checked. If there are two or more different colleges or school departments involved, we provide the retrieval results to three experts, and if they further reach a consensus, the evaluated problem is considered a multidisciplinary problem; otherwise, it is considered a single disciplinary problem. For example, the research problem “heat transfer, melt, calculation” showed significant discrepancies in manual classification. However, upon searching the CNKI database, the top five relevant master’s theses were affiliated with the following school department information: “School of Mechanical Engineering with a focus on energetic materials processing,” “Power Engineering and Engineering Thermophysics,” “Architecture and Civil Engineering,” “Energy Power and Mechanical Engineering,” “Power Engineering and Engineering Thermophysics.” The search results were sent to experts who agreed that the problem was multidisciplinary. Therefore, it was ultimately considered that this research problem would fall into a multidisciplinary field.

The results of the manual evaluation are shown in Table 7, and the accuracy of the identified multidisciplinary research problems was found to be approximately 68%. However, manual analysis is affected by the scope of individual knowledge, and some of the problems that are judged to be single discipline may be multidisciplinary problems. For example, the research problem “community, microorganism, carbon source” which may seem like a singular research problem in the field of Environmental Science and Engineering, is closely related to Agricultural Engineering as well.

An analysis of the disciplines involved in multidisciplinary research problems was conducted, and a disciplinary distribution chart for multidisciplinary problems was created, as shown in Figure 3. Region 1 of the figures displays the number of multidisciplinary research problems under each first-level discipline. Regions 2 and 3 together illustrate the intersection of multidisciplinary research problems across different disciplines. Region 2 represents the distribution of first-level disciplines within the intersection, whereas Region 3 represents the distribution of the number of problems within the intersection. The Venn diagram in Region 4 clearly illustrates the intersection of research problems across different first-level disciplines.

Figure 3.

It was found that disciplines “0817 Chemical Engineering and Technology” and “0830 Environmental Science and Engineering” have the highest number of multidisciplinary research problems. This indicates that in the field of CPCN research, these two disciplines have broad research scopes, or their research results are widely applied in other disciplines. It can be observed that there are three multidisciplinary communities in the field of CPCN. The first community consisted of “0703 Chemistry”, “0817 Chemical Engineering and Technology”, and “0820 Oil and Gas Engineering”. The second community includes “0805 Materials Science and Engineering”, “0813 Architecture”, “0828 Agricultural Engineering”, and “0830 Environmental Science and Engineering”. The third community comprises “0807 Power Engineering and Engineering Thermophysics”, “0819 Mining Engineering”, and “0823 Transportation Engineering”.