Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions—A Trial Dataset

Annotating structured contributions information from scholarly articles (see Section 4 for details), showed that, per article, its contribution-centered content could be organized as three or more different rhetorical categories inspired from scholarly article section names. Specifically, 12 different contribution content types were identified, a few of which were in common with the fine-grained rhetorical semantic classes annotations made for scholarly article sentences (Teufel, Siddharthan, & Batchelor, 2009). The 12 types are as follows.

i. ResearchProblem determines the research challenge addressed by a contribution using the predicate has Research Problem. By definition, it is the focus of the research investigation; in other words, the issue for which the solution must be obtained. It is typically found in an article's Title, Abstract, and in the first few paragraphs of the Introduction.

ii. Approach or iii. Model This is the contribution of the paper as the solution proposed for the research problem. The choice of the name, i.e. whether it is Approach or Model, depends on the paper's content. As a guiding rule of thumb, it is called Approach when the solution is proposed as an abstraction, and is called Model if the solution is proposed in practical implementation terms. Further, in the paper, the solution may not always be introduced as approach or model; in which case, the names have to be normalized to either Approach or Model. E.g. if the solution is introduced as “method” or “application,” this is normalized as Approach; on the other hand, if it is referred to as “system” or “architecture,” it is normalized to Model. In terms of the content captured, only the highlights of the solution are relevant in our contribution-focused model—in-depth details such as architectural figures or equations are not included. This information is found in the article's Introduction section in the context of cue phrases such as “we take the approach,” “we propose the model,” “our system architecture,” or “the method proposed in this paper.” However, there are exceptions when the Introduction does not present an overview of the system, in which case we analyze the first few lines of the system description section itself in the article. The Approach or Model is connected to the Contribution root node via the predicate has—this is followed for the remaining nine units as well.

iv. Code is a link to the software on open-source hosting platforms such as Github or Gitlab, or even on the author's website. This information, when present, can generally be found in the Abstract or Introduction, and occasionally in the Methods or Conclusion sections.

v. Dataset is sometimes included as a contribution information unit when it is a solution proposed in an NLP article. E.g. the SQuAD dataset that defines a Question Answering task.

vi. ExperimentalSetup or vii. Hyperparameters include hardware (e.g. GPU) and software (e.g. Tensorflow

https://www.tensorflow.org/

viii. Baselines are systems that the proposed Approach or Model is compared against. It structures information about which existing systems the proposed model is compared to.

ix. Results are the main findings or outcomes reported in the article for the ResearchProblem. This content is generally found toward the end of the article in the Results, Experiments, or Tasks sections. While the results are often highlighted in the Introduction, unlike the Approach or Model units, in this case, we annotate the dedicated, detailed section on Results because results constitute a primary aspect of the contribution.

Next, we discuss the 10^th IU called Tasks which can either be encapsulated in the Results IU or which encapsulates several Results IU against different tasks.

x. Tasks in multi-task settings, i.e. when the solution is tested on more than one task, list all the experimented tasks. Similarly, if experimental datasets are listed, they can be interpreted as tasks since it is common in NLP for tasks to be defined over datasets. Each task listed in Tasks can include one or more of the ExperimentalSetup, Hyperparameters, and Results as sub-information units.

xi. Experiments is an encapsulating IU with one or more of the above five results-focused IUs. It can include a combination of ExperimentalSetup, lists of Tasks and their Results, or can be a combination of Approach, ExperimentalSetup and Results. It resembles the scholarly article sections “experiment” or “experiments.” Recently, more and more multitask systems are being developed, e.g. BERT (Devlin et al., 2018). Therefore, modeling ExperimentalSetup with Tasks and Results is necessary for such systems since the experimental setup often changes per task producing a different set of results. Hence, this information unit encompassing two or more sub information units is part of the NCG scheme.

xii. AblationAnalysis is a type of Results that shows in fine-grained detail the contribution of each component in multi-component systems proposed for NLP tasks as the Approach or Model. In some articles, an ablation study is conducted to evaluate the performance of a system by removing certain components, to understand the contribution of the component to the overall system. Therefore, this IU is used to model these results if they are expressed in a few sentences, similar to our modeling criteria for Hyperparameters.

This concludes the detailed description about the 12 top-level IU nodes in the NCG scheme. Of the 12, only three are mandatory for structuring contributions per scholarly article. They are: ResearchProblem, Approach or Model, and Result—the remaining may or may not be present based on the content of the article.

The pilot annotation task (D’Souza & Auer, 2020) that resulted in the preliminary version of the NCG scheme was performed on a set of 50 NLP scholarly articles. This set of 50 articles constituted the trial dataset. Thus, the preliminary NCG scheme was defined over the trial dataset in a data-driven manner during the pilot annotation exercise.

There were two requirements decided at the outset of the annotation task. First and foremost, the graph model needed to be a robust representation for the diversity in NLP scholarly contributions. To facilitate this, a representative dataset of the NLP domain was needed so that unique aspects of contributions could be observed. Thus, the articles were uniformly chosen across five different NLP subfields: 1. machine translation, 2. named entity extraction, 3. question answering, 4. relation classification, and 5. text classification.

These five tasks were randomly selected among the most popular NLP tasks on the paperswithcode.com leaderboard.

The second design choice, regards the granularity of the data annotated in the NCG scheme. In the Related Work (Section 2), we saw that the sentential and phrasal granularity was used in prior work on structuring scholarly articles. Thus, inspired from this prior annotation science and also toward modeling KGs, the following three different granularity levels were established. At the first level, sentences that capture information about an article's contribution; at the second level, phrases about scientific knowledge terms and relation predicates identified from the selected contribution-information sentences; and, at the third level, toward building KGs, triples uniting the scientific terms and predicates as KG instances.

Table 1 below illustrates two examples of modeling contribution-oriented sentences, phrases, and triples from a scholarly publications categorized under the Results IU. Examples 1a and 2a, respectively, first show the sentences information granularity in the NCG scheme. These sentences discuss performance comparisons of the proposed model with existing scores. In general, for the Results IU, similar sentences that discuss performance comparisons of the proposed system versus existing systems should be selected as NCG model instances. Next, as examples of the phrasal granularity, 1b and 2b reflect the scientific knowledge terms and predicates from the sentences 1a and 2a, respectively. These are selected such that they should be able to take one of three roles, i.e. subject, predicate, or object, in triples for building KGs. Linguistically, they span a diverse spectrum of types, i.e. they can be noun phrases or verb phrases or adjectival phrases, etc. Further, the types are not related to the triple roles. Consider from 1b, the verb phrase “adding features” or the noun phrase “neural networks” can be either subject or object, whereas the phrase “computed by” which is the beginning part of a verb phrase is a predicate indicating a relation. Last, 1c and 2c illustrates the arrangement of the phrases as subject-predicate-object KG triples. Note, however, the order of the annotation steps need not be the same as the examples. At the least, we imagine sentences necessarily are first in the annotation task order. Following which, the annotator can perform the triples annotation first and later break their structure into phrases or the reverse. In the reverse case, they would make implicit triples considerations. In this context, one might wonder why include the phrases data element at all as part of the NCG scheme. We do this to enable flexibility in problem formulation for machine learning and, more concretely, to enable a clearer evaluation scenario since it could be that a machine learner may not be very accurate in forming triples but is better at identifying phrases.

We refer the reader to our prior work (D’Souza & Auer, 2020) for additional details regarding the pilot annotation task itself.

We carried out a two-stage annotation cycle over the trial dataset to finalize the NCG scheme. The first stage was the pilot annotation cycle with a brief overview provided in the earlier section (Section 4.1). The second stage was the adjudication cycle during which the graph model was finalized and the gold-standard article-wise graph data were obtained. A gap of two weeks separated the pilot and adjudication stages. Since the model development and annotation exercise was performed by a single annotator—an NLP domain specialist—, the stages needed to be sufficiently apart in time to control for the decision-making thought process from the pilot stage influencing adjudication. Nonetheless, the second stage was mere adjudication, therefore was non-blind. This meant that the annotator has access to the pilot annotations when re-annotating the data. Our adjudication objective was not to arrive at a new model that significantly differed from the model in the pilot stage, but it was to normalize that model itself. In other words, by adjudications, we merely sought to refine the quality of the scheme. Given the complex nature of this annotation task (cf. annotation evaluations in Section 5), and the need to design a model within a realistic timeframe, our annotation procedure is well-suited. Conversely, adopting a different annotation procedure, i.e. perhaps a blind second stage, might have resulted in a model with no consensus at all, since a consensus would have taken an unrealistic amount of time to arrive at—consider that since the NCG scheme is designed over the full-text of the scholarly articles its data consideration scope is wide despite the information constraint of the IUs. Further, using a different annotator for the adjudication stage than in the pilot stage could potentially result in a new model if the two annotators had different interpretations of contribution information—neither being incorrect, just different. This is a common phenomenon observed in the ontology engineering community where several ontologies exist under one knowledge theme but still modeling different worldviews. We also advocate for a blind, multi-stage, and multi-annotator annotation process for the NCG scheme, recognizing it as a potentially better annotation model since it can produce a well-defined set of annotation rules based on strict linguistic patterns agreed upon by multiple views. However, this is relegated as future work with a stipulated time-frame of at least two years. On the other hand, our consensus-oriented method is practically better suited as a first step toward designing complex graph-based models, in other words, to obtain the NCG scheme in a realistic timeframe.

There were two requirements decided at the outset of the adjudication annotation task. They were: 1) to normalize IUs further to be a smaller, but comprehensively representative, set of similar structured properties to facilitate succinct contribution comparisons across articles’ contribution graphs; and 2) to improve the phrasal boundary decisions made in the pilot stage focused on targeting precise scientific knowledge semantics within the annotated phrases. Otherwise, both the pilot and adjudication stages adopted the same annotation workflow as depicted in Figure 2.

Figure 2

Let us elaborate on the two requirements: 1) normalizing IUs—we had 16 different IUs in the pilot stage. During the course of the adjudication stage annotations, they were normalized as the following nine IUs: ResearchProblem, ExperimentalSetup, Hyperparameters, Results, Tasks, Experiments, AblationAnalysis, Baselines, and Code were retained as is; the set Model, Approach, Method, Architecture, System, and Application was reduced to Model and Approach—in our second round of observations of the data, we identified references to Method, Architecture, System, and Application all pointed at an actual piece of software rather than a conceptual solution, thus they were normalized into a single Model IU, while Approach referred to the conceptual solution; and the Objective IU was dropped since it was infrequent. Further, a Dataset IU was added, since in NLP papers they are sometimes described as a solution. 2) improving phrasal boundaries—this involved making smaller split decisions on phrases to convert very specific scientific knowledge elements into entities that could take on more generic roles and therefore were reusable in a KG. E.g. “models that use extensive sets of handcrafted features” split into the following three phrases “models,” “use,” and “extensive sets of handcrafted features.” This example is depicted in detail in Appendix 2.

4.2.1

The NCG Scheme's five general annotation guidelines

Finally, five main annotation guidelines are prescribed for NCG.

1)

Sentences with contribution data are identified in various places in the paper including title, abstract, and full-text. Within the full-text, only the Introduction and Results sections are annotated. Sometimes, the first few sentences in the Methods section are annotated as well if method highlights are unspecified in the Introduction.

2)

Only sentences that directly state the paper's contribution are annotated.

3)

All relation predicates are annotated from the paper's text, except the following three, i.e. has, name, and hasAcronym. “has” is used for connecting the IU nodes to the Contribution root node, and as a filler predicate if one is not found directly in the text. “name” is the filler predicate used to link the model name to the Approach or Model nodes. “hasAcronym” has a similar function to name, except applied only for acronyms of names.

4)

Past the IU level, for parent node names in the graph, the names of sections are preferred, which if challenging to annotate are identified in the running text (see Appendix 1 example).

5)

Repetitions of the scientific terms or predicates, which do not correspond to the actual information in the text, are not allowed when forming KG triples. See illustrated in Figure 3.

Figure 3

The trial dataset for designing the NCG scheme was derived from a collection downloaded from the publicly available leaderboard of tasks in artificial intelligence called https://paperswithcode.com/. The paperswithcode.com dataset predominantly contained articles in the Natural Language Processing and Computer Vision domains. To design the NCG scheme, we restricted ourselves just to its NLP papers. From the original collection of papers, we randomly selected 10 papers from each of the five different NLP subfields which resulted in a total of 50 papers. The five NLP subfields were: 1. machine translation, 2. named entity recognition, 3. question answering, 4. relation classification, and 5. text classification.

The raw data needed to undergo a two-step preprocessing to be ready for analysis. 1) For pdf-to-text conversion of the scholarly articles, the GROBID parser (GROBID, 2008) was applied; following which, 2) for plaintext pre-processing in terms of tokenization and sentence splitting, the Stanza toolkit (Qi et al., 2020) was used. The resulting pre-processed articles were then leveraged in the two-stage annotation cycle (see Section 4) to devise the NCG scheme.

The overall annotated corpus statistics for our trial dataset after the adjudication stage is depicted in Table 2. We see that, in each of the five subfields, approx. 40 IUs were annotated on each of the subfield's articles, i.e. on average four IUs per article per subfield since 10 articles were selected for each subfield. This implies that, on average, each article was annotated with more or less one additional IU beside the three (ResearchProblem, Approach or Model, and Results) deemed mandatory. Further, in the annotated data, per subfield, we see at the sentence element level, that a very small fraction (< 0.1%) of the scholarly article sentences constituted contribution sentences (see Table row “avg ann Sentences”). In terms of tokens, < 0.3% of the overall tokens were annotated as scientific terms or predicates toward KG building (see Table row “avg ann Phrase Toks”). Finally, in terms of triples, approx. 600 triples were annotated per task, i.e. nearly 60 triples per article per task. Thus, overall, while not much data is annotated per article (approx. 16 sentences and 90 phrases per article—see Table rows “ann Sentences” and “ann Phrases” for the total counts), the task challenge is selecting the pertinent contribution-focused information given the wide range of the available candidate data.

Next we highlight a few differences between the five subfields in our dataset. In Table 2, we see that machine translation (MT) had the most annotated sentences. This can be attributed to the observation that generally MT articles tend to be longer descriptively and particularly in terms of models settings compared to the other four subfields. Further, relation classification (RC) had the highest proportion of contribution sentences constituting its articles. With 0.1 this still indicates a low proportion reflecting the fact that contribution information is contained in relatively few sentences. Text classification (TC) had the highest number of annotated phrases despite not being among the tasks with the highest numbers of annotated sentences. This implies that the number of annotated sentences is not directly related to the number of annotated phrases for the tasks in our data. But this understandably is not the same concerning the correlation between the number of phrases and triples, wherein the number of phrases and triples are directly related.

Table 3 depicts the final annotated corpus statistics in terms of the information units. We make the following key observations: Results has the most number of triples; ResearchProblem is modeled for all articles; and Experiments has the highest triples-to-papers ratio. We see that Experiments is annotated only in three articles yet contains a high number of triples. This can be attributed to the fact that this particular IU can also encapsulate other IUs such as ExperimentalSetup, Tasks, and Results (cf. Section 3.1). This is different from the other IUs that have no other IU encapsulation, with Tasks as an exception.

We now compute the intra-annotation agreement measures between the first and the second stage versions of the dataset annotations across all three data elements in the NCG scheme including its top-level information units. Our evaluation metrics are the standard precision, recall, and F1-score.

Table 4 depicts the results. With these scores, we quantitatively observe the degree of changes between the two annotation stages treating the second stage as reference gold-standard. Between the two stages, the F1-scores of the annotation changes were: information units 79.64%, sentences 67.92%, phrases 41.82%, and triples 22.31%. We conclude that the interpretation of annotations related to the top-level organization of scholarly contributions did not change significantly (at 76.64% F1-score). Even the decision of the annotator about the sentences containing contribution-centered information showed a low degree of change (at 67.92% F1-score). However, the comparison of the fine-grained organization of the contribution-focused information, as phrases or triples, obtained low F1-socres. Finally, from the results, we see that our pipelined annotation task is presented with the general disadvantage of pipelined systems, wherein the performances in later annotation stages is limited by performances in earlier annotation stages.

In light of the low intra-annotator agreement obtained at the phrasal and triples information granularities, a natural question may arise: is our proposed model valid at all? We claim validity on the following basis. 1) Since the data was annotated by a single person, there is a data modeling uniformity in their decisions arising from the fact that it was made by the same person. Thus, the annotated data in any stage consistently models the annotator decisions in that stage. For instance, regarding the sentences, major discrepancy arose between the pilot and the adjudication stages from decisions such as selecting titles as contribution sentence candidates, wherein in the first annotation stage, titles were not annotated, but were uniformly selected to be annotated in the adjudication stage. 2) Further, as shown in the “avg ann Sentences” row in Table 2, since roughly only 8% of the article sentences were contribution sentences, an expected consequence of this task is indeed a high amount of variance in the sentence selection decisions. Such a discrepancy can only be resolved with further annotation stages. With reference to our earlier example, since titles were uniformly established as contribution sentence candidates in the second annotation round, in a third round we would have much higher modeling decision scores. 3) Finally, our annotation task has a pipelined nature. Thus, sentence modeling discrepancies are magnified in the later annotation stages, i.e. for phrases or triples annotations. This explains a major proportion of the low agreement scores in the second annotation round for phrases and triples since new phrases (and triples) had to be annotated for the new added sentences.

Observing the task-specific intra-annotation measures in rows 1 to 5, we find that question answering (QA) and text classification (TC) have the highest scores reflecting fewer changes made in their annotations than the other tasks, albeit at decreasing levels across the data elements.

Generally, one may wonder why triples formation is challenging given a set of scientific term and predicate phrases indicated by its lowest F1-score per task and overall. There are two reasons: a) the phrases were significantly changed in the second stage (see Phrases F1-scores as < 50%), which in turn impacted the triple formation; and b) often the triples can be formed in more than one way. In the adjudication process, it was established for the triples to strictly conform to order of appearance of phrases in the text, which in the first stage was not consistently followed.

The ORKG comprises structured descriptions of research contributions per article. The user can enter the relevant data about their papers via the framework online at https://orkg.org. In Figure 4 below, we depict our annotated triples in plain text format for the Results IU capturing just the results contribution aspect of a paper. Note that this data can alternatively be represented in JSON format (as in Figure 3 shown earlier in this paper).

Figure 4

With the help of the ORKG paper editor interface, we add a paper to the ORKG including its bibliographic information, the Results IU data (Figure 4), and the ResearchProblem IU data. This paper view in the ORKG is depicted in Figure 5 below. In the figure, while the data values for ResearchProblem IU in the NCG scheme is directly visible, the data for the Results is not. This is due to the paper-view showing only the top-level node. To access deeper levels of the structured data, one would need to click on the Results text link in orange. Such a branch traversal starting at Results until the last node is depicted in Figure 6 as a four-part series.

Figure 5

Figure 6

With this we have described how the structured contributions data of individual papers are represented in the ORKG. Next, we showcase the ORKG feature for generating comparisons.

The ORKG has a feature to generate and publish surveys in the form of tabulated comparisons over articles’ knowledge graph nodes (Oelen et al., 2020). To demonstrate this feature, we entered our data for the Results IUs of four papers including the one depicted in Figure 5 in the ORKG. Applying then the ORKG survey feature on the four structured articles contributions aggregates their semantified graph nodes in a tabulated comparison across them. This is depicted in Figure 7. This computation aligns closely with the notion of traditional survey articles, except it is automated and operates on machine-actionable knowledge elements. With such an aggregate view over common contribution properties it is easier for users to ingest the essential details of the articles in a matter of minutes in a single view, a feature that is even more advantageous when a larger number of articles is compared. Given that the data is directly extracted from the text it is amenable to further processing for better knowledge representation quality (e.g. “Outperforming,” “Outperforms,” and “Significantly outperforms” can be normalized as a single predicate).

Figure 7

Thus we have demonstrated how structured contributions from the NCG scheme address the massive scholarly knowledge content ingestion problem.