Extracting and Measuring Uncertain Biomedical Knowledge from Scientific Statements

Computable knowledge must be structured, or machine-readable. A simple representation of computable knowledge is the SPO triple, which consists of a subject argument, an object argument, and the predicate that binds them. For instance, drug knowledge can be expressed as triples (Elkin et al., 2011). Based on the comparison between triples from PubMed articles and the Food and Drug Administration (FDA) drug labeling documents, one can identify novel and unreported medical knowledge in the literature (Malec & Boyce, 2020).

The SemRep tool and SemMedDB knowledge database, which were developed and maintained by the Semantic Knowledge Representation program of the National Library of Medicine (NLM), are typical representatives for SPO triple extraction from biomedical free text. SemRep, the abbreviation for Semantic Representation, is a rule-based natural language processing tool that extracts SPO triples from sentences in biomedical texts. It is based on standardized medical concepts, concept types, and semantic relations between concepts in the Unified Medical Language System (UMLS). The latest 2021 version of UMLS contains approximately 4.5 million concepts, 127 concept types, and 54 semantic relations. SemMedDB, short for Semantic MEDLINE Database, is a repository consisting of SPO triples extracted from the titles and abstracts of all of the PubMed publications (Kilicoglu et al., 2012) by SemRep (Rindflesch & Fiszman, 2003). The SemMedDB also stores SPO triple-related information, including the supporting sentences as well as the original publication (Kilicoglu et al., 2012). NLM regularly releases a new version of SemMedDB with improvement, including modifications of incorrect concepts and relations extracted by the SemRep, which is currently being redesigned to improve its overall performance (Kilicoglu et al., 2020). SemRep and SemMedDB form a fundamental database that realizes the extraction and storage of large-scale knowledge units, supports secondary knowledge discovery, and supports a variety of clinical decision-making and applications to help improve health outcomes. For example, the MELODI Presto system (http://melodipresto.mercies.ac.uk) has recently been developed for web-based SemMedDB SPO triple and metadata queries (Elsworth & Gaunt, 2021).

In recent years, progress in the field of biosemantics has provided solutions for the representation of biomedical knowledge. As Barend Mons and other scientists proposed (Groth, Gibson, & Velterop, 2010; Mons et al., 2011), the basic structure of the nanopublication model includes the following components: a) assertion, the scientific claims represented by SPO triples; b) provenance, referring to the author, institution, time, position, and other necessary information of a triple; and c) publication information, or the metadata of a nanopublication itself, including the creator, date of creation, and version of a nanopublication. The three components are indispensable to ensure the integrity of the information and effectively improve the possibility of reusing scientific research information. Expressed in RDF format, the contents of these components are machine-readable. Surveys in the biomedical domain indicate that there are approximately 10¹⁴ such assertions, with an enormous increase every year. After eliminating duplicate findings, 10¹¹ canonical assertions still remain. Further processing yields approximately 10⁶ so-called knowlets, which can be seen as core concepts of assertions related to sets of different findings (Mons, 2019), and they can be used to draw conclusions or for proper health treatments.

Nanopublication has been applied for scientific research. Currently, converting existing relational databases (such as DisGeNet, a gene-disease association knowledge database) to nanopublications (Fabris, Kuhn, & Silvello, 2020) or storing and computing triples at online platforms such as The Open Pharmaceutical Triple Store (Open PHACTS, a pharmaceutical semantic triple discovery platform) (Williams et al., 2012) are two representatives of nanopublication applications. However, the nanopublication model has not been widely utilized in clinical research and practice. To promote the reuse of nanopublications and increase acceptance, scholars have recently proposed a nanocitation format that can cite individual nanopublications and allocate credits to the contributor. Therefore, it is possible to design bibliometric indicators, conduct fine-grained analysis of knowledge units, and combine them with current scientific research (Fabris, Kuhn, & Silvello, 2019).

Barend Mons (2019) also suggested aggregating common assertions that appear in all nanopublications into a cardinal assertion for presenting a basic knowledge unit to reduce redundancy. This process can be considered the formation of a knowlet, which combines all of the directly linked concepts around a central concept. We think that the knowlet, or the knowledge subgraph, can represent an independent knowledge unit in the knowledge graph. Compared with the rapid growth of support text related to a particular assertion, the growth of the knowledge subgraph is relatively slower. For instance, a large amount of text may involve only one piece of knowledge unit. The knowledge subgraph is an independent digital object, and the smallest knowledge unit can be discoverable, accessible, interoperable, and reusable.

Uncertainty is an inherent part of knowledge. It is a rhetorical means, or a communicative technique, to convey the degree of uncertainty associated with a statement or an assertion (Chen & Song, 2017). Understanding the wide variety of types and the degree of uncertainties in science and their roles in the advancement of science itself and in the broader context of everyday life is the first step toward a better understanding of how science works (Chen & Song, 2017). The uncertainty associated with a scientific proposition implies the epistemic status of scientific knowledge, which is easily neglected meta-knowledge. Pluralistic epistemic statuses may indicate the emergence of a new paradigm. The degree of uncertainty may affect the attentions to scientific research because it may imply different levels of achievements.

Kilicoglu, Rosemblat, and Rindflesch (2017) annotated semantic predications with seven factuality values (fact, probable, possible, doubtful, counterfact, uncommitted, and conditional), and extended a rule-based, compositional approach that uses lexical and syntactic information to predict factuality levels. Uncertainty cues are a particularly relevant traits in understanding how scientists characterize the tentative and context-dependent nature of scientific claims (Chen & Song, 2017). Szarvas et al. (2012) identified uncertainty cues in each of these categories. For instance, “probably,” “likely,” and “possible” are uncertainty cues in the adjective and adverb category. Examples of auxiliaries as uncertain cues include “may,” “might,” and “could.” Speculative verbs include “suggest,” “seem,” and “appear.” Nouns include “speculation,” “proposal,” and “rumor.” In scientific writing, uncertainty cues come from adjectives, adverbs, auxiliaries, verbs, conjunctions, and nouns in general.

In this article, by combining informetric methods with computational linguistics, we aimed to extract knowledge units from scientific statements in cardiovascular research publications in China. In our previous work, we proposed a framework for medical knowmetrics using SPO triples as the knowledge units and uncertainty as the knowledge context. However, it is not well resolved how to measure the uncertainty level for a given SPO triple or concept pairs. Therefore, inspired by Chen's work (2020), we introduced the concept of information entropy (IE) to measure the level of uncertainty for a given knowledge unit.

For constructing the dataset of structured biomedical knowledge from scientific literature, the process consists of collecting related publications, extracting structured semantic predications (SPO triples) from those publications, and classifying semantic types and relations for the extracted SPO triples.

A total of 17,594 English articles and 17,196 Chinese articles published between 2001 and 2020 satisfied our specification. After excluding Chinese articles without English abstracts, articles with multiple PMIDs, and entirely equivalent SPO triples extracted from the same sentence, we obtained 29,800 articles, 148,409 unique sentences, and 259,067 predications. Among them, a total of 100,103 unique triples were related to 17,829 concepts, which were grouped into 117 semantic types and then 15 semantic groups; 61 semantic relationships were grouped into 8 categories, and we constructed the dataset with relevant information. For identifying overall evolution of cardiovascular research in China, Figure 2 presents the basic statistics of the articles and the triples in our dataset. Over a 20-year period, extraordinary progress of cardiovascular research in China was observed. The number of total SPO triples parsed from all articles has significantly enlarged year over year, reaching 20,660 by the end of 2019, which indicates an increase by 6.1 times. It was almost consistent with five-fold growth of the number of publications. In contrast, the number of novel triples (the newly occurring triples) increased by 2.9 times since 2001 with modest growth (Figure 2A).

Figure 2

For discovering the evolution of topics in structured biomedical knowledge over time, we examined the concepts and relations in SPO triples. A total of 17,829 semantic concepts and 117 semantic relationships were clustered into 15 semantic groups and eight categories, respectively. As shown in Figure 2B, “Disorder” was the leading area growing over time (top, dark blue). Concepts within this semantic group included “Hypertensive disease,” “Cerebrovascular accident,” “Coronary arteriosclerosis,” “Coronary heart disease,” and “Atrial Fibrillation.” The proportion of “Chemicals & Drugs,” such as “Serum,” “Aspirin,” and “Glucose,” showed a relative decline, although their number increased gradually. Among the eight semantic relation categories, “functionally_related_to” (such as “PROCESS_OF,” “CAUSES,” and “AFFECTS”) accounted for the largest proportion, with up to 60% with significant growth. In contrast, the proportion of “spatially_related_to” decreased from 20% in 2001 to 15% in 2019. Other groups were relatively small and remained stable over the years (Figure 2C).

To describe the core dataset, we provide SPO triples extracted from cardiovascular literature and relevant information in Table 2. We found the following: (1) examples of #1, #4, #5, and #6 were typical representatives of scientific assertions reconstructing biological knowledge in the literature; (2) some triples were more likely to be general concepts or facts rather than scientific claims, i.e., “Stem cells_Part_of_Marrow” standing for “MSCs (mesenchymal stem cells)” in #3; (3) multiple triples extracted from a given sentence could express richer semantic information, in spite of missing some semantics in #2; and (4) different predicates in the triples of same subjects and same objects tended to present similar or progressive viewpoints, such as #4 and #5.

Considering the topic mainly determined by concepts, we identified and measured uncertain biological knowledge at a subject–object pair (SO pair) level in the following study. Therefore, IE was applied to measure the uncertainty of each SO pair. The uncertain knowledge with the highest IE was related to the SO pairs in Table 3, in which the number of supporting sentences is no less than 4. The following information indicated disease-centric uncertain knowledge since our dataset involved cardiovascular research publications. We found that the uncertainty in “genetic function,” “fibrinogen,” and “epinephrine,” related to cardiovascular research in China remained at a high level in general. All of these emerged after 2000, and the number of their supporting sentences was not more than 15. The epistemic statuses of scientific knowledge in these topics are pluralistical, implying its active role in cardiovascular research.

We examined the supporting sentences of the SO pairs with high IE, and details are presented in Table 4. Overall, a given SPO triple extracted from biomedical literature was only a frame part of propositions. As for “Polymorphism, Genetic_Coronary Arteriosclerosis,” the scientific statements concerned different genes, i.e., “receptor for advanced glycation end product (RAGE),” “myocyte enhancer factor 2A (MEF2A),” “renin–angiotensin system (RAS),” “fibroblast growth factor receptor 4 (FGFR-4),” “myeloperoxidase (MPO),” “nitric oxide synthase (NOS3),” and “interleukin-8 (IL-8)”. In “Hydroxymethylglutaryl-CoA Reductase Inhibitors_Acute coronary syndrome,” the SO pair was not representative of specific conditions, such as “intensive lipid lowering,” “advanced age,” “percutaneous coronary intervention (PCI),” “early use,” “combined treatment,” and “use of prehospital.” In “Hyperuricemia_Hypertensive disease,” “PREDISPOSES” reflects that “hyperuricemia is a risk factor for hypertension” appropriately. However, there was hardly any regularity of epistemic status evolution in terms of uncertainty type. When considering various research differentiation in a given topic, the IE in this article did not assign factuality values to the scientific assertion, but rather indicated epistemic status pluralism of the SO pair, which implied its activity in the area.

For a visual and comprehensive overview, we prepared a map of SO pairs and their interrelation based on co-occurrence within publications using a network analysis tool, VOSviewer (Figure 3). The nodes weighted by occurrence were top 1000 SO pairs with greatest total link strength, following exclusion of “patients” and “population group” among objects. In Figure 3A, the clusters are depicted in different colors, representing similarity of research, clinical (center, red/pink/dark green), population (left, green/blue), and basic research (right, orange/yellow). Large nodes in seven clusters suggested the following major research directions: (1) “implantation procedure_stents” and “percutaneous coronary intervention_acute myocardial infarction” in clinical research of surgery (red); (2) “therapeutic procedure_pharmaceutical preparations” in the study of drug intervention (pink); (3) “mucocutaneous lymph node syndrome_child” and “congenital heart disease_child” in clinical research on childhood-onset diseases (dark green); (4) “plasma_glucose” and “hypertensive diseases_adult” in population research on cardiovascular risk factors (green); (5) “patients_single nucleotide polymorphism” in the genetic polymorphism study (blue); (6) “myocytes, cardiac_apoptosis” in basic research of heart diseases (orange); and (7) “cerebral ischemia_rattus norvegicus” in basic research of cerebrovascular diseases (yellow).

Figure 3

Biological knowledge with uncertainty value is presented in Figure 3B. Cool–warm colors representing IE were overlayed on each node. The configuration of the parameters for this network is shown in the bottom right corner. Redder nodes reflect more uncertainty types, implying hotter research topics. In clinical research, cerebrovascular accident, atrial fibrillation, and congenital heart diseases were active research points. In population research, co-existing risk factors related to metabolic syndrome, such as hypertensive disease, diabetes, obesity, and hyperuricemia, were prominent. Research on genetic polymorphism was considered an emerging field in the past decade, presenting the highest uncertainty in seven clusters. Basic research was another active area with high IE; it included cardiac myocytes apoptosis in reperfusion injury and electroacupuncture against cerebral ischemia.

We introduced the concept of knowledge unit, defined as one identical SPO triple derived from multiple source sentences. The SPO triple is a structured representation for knowledge assertions, and is less abundant than the unstructured text. We observed a less steep development of novel SPO triples compared with scientific publications and total SPO triples, which suggests that new knowledge or research fields are not growing as fast as the number of scientific publications. This outcome is in accordance with the results of “overall growth of science” within all publications in Web of Science. Although the number of publications grows exponentially, the conceptual territory of science, measured by the newly appeared phases in the title, expands linearly (Fortunato et al., 2018).

We also added contextual uncertainty to SPO triples since they were often extracted from hypothetical, speculative, or even conflicting and contradictory assertions. Here, we considered a transparent methodology that used a set of cue words to measure the level of contextual uncertainty rather than a machine-learning approach. It is also consistent with a recent related work (Kilicoglu, Rosemblat, & Rindflesch, 2017), which investigated the feasibility of assessing the factuality level of SemRep predications. These authors annotated semantic predications extracted from 500 PubMed abstracts with seven factuality values (fact, probable, possible, doubtful, counterfact, uncommitted, and conditional). They proposed a rule-based, compositional approach that used lexical and syntactic information. This method was compared with a supervised machine-learning method that used a rich feature set based on the annotated corpus. The results indicated that the compositional approach was more effective than machine learning in predicting factuality levels. This transparent approach allows us to easily scale our analysis to millions of scientific articles while also being transparent and reproducible, and ensuring interpretability of the results.

Uncertainty cue words were applied to identify and measure the uncertainties in this study. The uncertainties of the context can be identified independent of the scientific assertions; however, it is more valuable to combine the explicit information from rhetorical cue words and tacit knowledge mined from SPO triples. In our current research, terms such as “action potential” and “as soon as possible” do not indicate epistemic status of knowledge. We will optimize the method and improve the specificity of uncertainty cue words. We may assign probabilities to cue words (i.e., the probability for “likely” is less than “very likely”). Future research could refine the method for measuring uncertainty by creating metrics for the uncertain degree of cue words.

Scientific knowledge is a complex adaptive system of facts, beliefs, hypotheses, speculations, and opinions. Uncertainty cue words are used as a rhetorical device or a communicative technique to convey the degree of uncertainty associated with a statement or an assertion (Chen & Song, 2017).

Generally, the structured abstract of a given biomedical article includes background, objectives, methods, results, and conclusions, reflecting the process and elements of knowledge production. The information in background is usually the premise and basics of the scientific issues to be investigated; objectives tend to be unconfirmed hypotheses; results convey evidence to support the objectives; and conclusions are the knowledge claims, arguments, and assertions concluded through the investigation. We determined different parts of the scientific statements by machine identification combined with manual interpretation.

We analyzed the textual distribution in the abstract of three uncertain knowledge types. According to the typical structure of an abstract and the process of knowledge generation, we annotated 7,530 sentences randomly with four types: background, objectives, results, and conclusions. The frequency of uncertainty cue words in different parts of scientific statements (Figure 4) presented the different distribution of hedging and conflicting knowledge. Authors tended to use hedging lexicons in conclusive and purposeful sentences, whereas conflicting lexicons applied in background would act as premises of the investigated work.

Figure 4

Examples of uncertain sentences and triples from different parts of scientific statements are listed in Table 5. Based on the results we observed, authors put forward existing knowledge in the background, presumption intention in objectives, evidence supporting the views of paper in results, and scientific claims and presupposition in conclusions. This suggests that uncertain knowledge, especially conflicting knowledge, is usually the premise and the starting point of scientific research, which can be considered as cognitive uncertainty. Moreover, authors generally describe objects and conclusions with hedging lexicons, probably due to cautious and modest attitudes toward presupposition and scientific claims.

We demonstrated three major sources of uncertainty in scientific literature, namely, unknown, hedging and contradictory information. Different types of uncertainty and positions of uncertainty can indicate distinct information. Three types of uncertainty, i.e., unknown, hedging, and conflicting, as well as a set of cue words were proposed for classifying the level of uncertainty. Previous research has distinguished between two kinds of uncertainty, epistemic uncertainty about the past and present vs. uncertainty about the unknown future (van der Bles et al., 2019). This implies that hedging lexicons are associated with assumptions about the future and conflicting lexicons are associated with epistemic uncertainty. The distribution of hedging and conflicting lexicons in different parts of scientific statements provides potential quantitative evidence of uncertain knowledge classification.

In this study, we found that IE was related to the scale of the knowledge units. With the increase of the supporting sentences, IE decreased and approached a certain value, which is also the IE value (0.38) of total biological knowledge in our triple store. We speculate that with the increase of the number of supporting sentences, the probability distribution of uncertainty types will trend to be the total corpus (Figure 5). This is also the limitation of the measurement method in this paper. Considering that small-scale samples are prone to small probability events, the number of the supporting sentences of the top 10 SO pairs with the highest IE is less than 15. In the future, we will consider the factor of scale and revise the uncertainty value.

Figure 5

The current approach to measure uncertainty by introducing IE essentially reflected pluralistically epistemic status of scientific knowledge. The more types, the more average the type distribution, and the greater the IE. With the expansion of knowledge boundary, there are more unknowns. The significance of knowledge uncertainty may be different at different scales. For the specific scientific assertion, the uncertainty of knowledge decreases with the increase of credibility. For a topic or research area, knowledge may not be from uncertain to certain; actually, the discovery of a scientific problem, which attracts more attention, assumptions, and verification, drives the increase of contends in relevant research fields. At the meso or macro level, uncertainty may be a sign of prosperity in a given field of science. We plan to apply our model to a broader range of data sets of medical knowledge, such as RCT registry information provided by clinicaltrials.gov, to further explore the feasibility of uncertain knowledge computation using this model. We plan to leverage uncertainty-centric approaches to detect research fronts, evaluate academic contributions, and improve the efficacy of computable knowledge-driven decision support.

In conclusion, we provided a novel approach by combining natural language processing and computational linguistics with informetric methods to extract and measure uncertain knowledge from scientific statements. IE is a potential metric to quantify the uncertainty of epistemic status of scientific knowledge represented at different levels, such as SPO triples (micro level) and semantic type pairs (macro level).