A Bootstrapping-based Method to Automatically Identify Data-usage Statements in Publications

As words of the same type are contextually similar to each other, if we set a few words as the starting point for searching, accompanied by their common contextual features, we can identify more words that possess similar contextual features. By repeating these processes, we can find more and more words that are similar to each other. What is needed to achieve this goal is the selection of probable representative data_clue words as the initial seed words and data_patterns of these data_clues. In this case, an unsupervised bootstrapping method is proposed (Figure 1), which is applied to acquire a set of data_clue and data_pattern pairs sharing similar features, i.e. a list of < data_clue, data_pattern >.

Figure 1

Seed-word selection is the only process that requires manual intervention in our method. The quality of seed words will directly affect the performance of the extraction method. In this paper, three seed-word selection strategies are chosen for implementation:

Selecting both the names of a few well-known datasets and a category of general indicative words, such as “dataset,” as seed words, referred to as COM-SEED;

In order to compare the performances of different strategies, we respectively conduct extraction experiments with each of the three different strategies for seed-word selection.

A pattern is used to describe the structural features of the target sentence to be extracted. Normally, the stronger the generalizability of a pattern, the wider the scope of the sentences covered by it; conversely, the stronger the representativeness of a pattern, the narrower the scope of the sentences covered by it. Synthetically, considering both the generalizability and representativeness of patterns, we believe that components of a pattern should at least include the core part of a complete sentence, i.e. the predicate part. Meanwhile, given that the seed words, which are almost exclusively nouns or noun phrases, usually occur in the subject or object part, we choose to construct two types of patterns:

Subject part + predicate part, particularly dealing with circumstances in which seed words occur in the object part;

Table 2 provides some examples of the two types of patterns and presents their extraction performance.

Table 2

Exemplifications of pattern construction.

In computer science, a single data-usage relationship pattern can be applied to varying data objects, e.g. different face-recognition datasets may all serve as a training dataset for machine learning in different articles. Moreover, a dataset entity can have multiple applications, e.g. the ClueWeb09 dataset has regularly been used for search-result ranking, query sub-topic mining, relevance evaluation of retrieval systems, and many other types of experiments. We therefore presume that any combination of one data_clue and one data_pattern adopted from their respective result list has the potential to be involved in a DUS. If a sentence contains at least one data_clue and at least one data_pattern belonging to its own result lists, it can be identified as a DUS.

In each iteration, a number of data_clue words and data_patterns will be, respectively, added to their corresponding final list if their own score has exceeded the current threshold. The score can be interpreted as the relative probability of a data_clue word or a data_pattern being regarded as valid, based on currently available evidence.

As illustrated in Figure 1, the bootstrapping process is triggered by adding the original seed words to the seed pool, and the procedures will be performed as below (define the current iteration as i, and the maximum number of iterations as MAX):

Obtain all of the patterns in the dataset of research papers;

After the bootstrapping process, a collection of < data_clue, data_pattern > pairs can be generated by randomly selecting a data_clue word from the final data_clue list, and meanwhile randomly selecting a data_pattern from the final data_pattern list to make a combination, which is referred to as “pair_set.” If any single sentence contains components that are in accordance with any certain pair existing in the pair_set, it is identified as a DUS.

(i) Evaluation measures: Random-selected samples of the experimental results are applied for evaluation in order to achieve a reasonable balance between the feasibility and reasonability of manual evaluation. We separately selected 300 statements extracted under the guideline of pairwise combinations of two seed-selection strategies and two pattern-construction strategies, which resulted in a total of 1,200 statements for manual judgment. The authors invited four master candidates in information science to make the judgments through tagging every statement according to whether or not it is within the realm of extraction. The operation instruction is as follows: “If a statement meets our extraction target, give a Yes tag; if not, give a No tag; and if it is not clear enough to determine, give an Unknown tag.” The majority of Unknown statements are those which only mention a dataset, but do not illuminate how the dataset relates to the corresponding article. To deal with all of the Unknown statements, the authors scrutinize all of the original full-text articles containing Unknown statements to make a final judgment of either Yes or No. The number of Yes statements is denoted by R, the total number of statements T, and the extraction precision is calculated as Equation (3): Precision=R/T.$$\begin{array}{} \displaystyle {\rm Precision} = R\,/\, T . \end{array} $$(3)

(ii) Results analysis: As can be seen from Table 5, different seed-selection strategies and pattern categories affect extraction precision to varying extent, in which both precisions under the pattern in the form of “Predicate + Object” are above 90%. Regardless of what pattern category is adopted, the COM-SEED strategy performs significantly better than GEN-SEED, in terms of precision.

Table 5

Precision of statement extraction from CSExperiment-triple (2000–2013).

Specifically, for different seed-selection strategies, the pattern form of “Predicate + Object” performs substantially better than that of “Subject + Predicate,” which is consistent with its structural properties as a human language. Assuming that we intend to generate a well-formed sentence, it will be much easier to seek out an eligible object for a given combination of subject and predicate than to seek out an eligible subject for a given combination of predicate and object. In other words, when the data_clue words are embedded in the subjective section and will be extracted through patterns in the form of “Predicate + Object,” the connection between them and the initial seed words is closer and more stable. Conversely, when the data_clue words are embedded in the objective section and extracted through patterns in the form of “Subject + Predicate,” the newly added data_clue word will be more prone to deviate from the scope of the initial seed words during the process of iteration.

Given the fact that the extraction precision under the COM-SEED strategy is much greater than that under the GEN-SEED strategy, it is logical to deduce that the specific dataset names added in the initial seed words will confine the contexts of candidate words within the target extraction range, which reduces noise caused by general indicative words, such as “data” or “dataset.”

(i) Evaluation measures: Extensibility of patterns is the effectiveness of extracting DUS from articles which do not belong to the original data collections, using patterns existing in the final pattern list which were previously acquired by the iterative process on the original article collection. The articles to be extracted include those within the same research field that do not exist in the original data collection, and articles which belong to other research fields. Taking the objective of the present paper into account, only extensibility within the same field is evaluated.

To evaluate the within-field extensibility of patterns, we randomly select 25 unique articles from the CSExperiment-triple (2014) data collection to create the evaluation dataset, which contains 2,015 sentences in total. A golden standard of 487 sentences concerning data-usage is generated through manual annotation from the evaluation dataset word-for-word. Any sentence which meets at least one data_ pattern in the final list is automatically extracted from the evaluation dataset to form the results collection, which are then compared with the golden standard.

Extensibility evaluation is achieved through comparing the results collection with the golden standard. Counted by sentences divided by periods, the number of all sentences in the results collection is denoted by R_n; the number of common sentences both in the results collection and in the golden standard is denoted by M_n; and the number of sentences of the golden standard is denoted by S_n. We use precision (accuracy of extension), recall (coverage of extension), and F-measure (overall extensibility) for evaluation. These three measures are calculated with Equations (4)–(6): Precision=MnRn,$$\begin{array}{} \displaystyle {\rm Precision} =\frac{M_{\rm n}}{R_{\rm n}}, \end{array} $$(4)Recall=MnSn,$$\begin{array}{} \displaystyle {\rm Recall} =\frac{M_{\rm n}}{S_{\rm n}}, \end{array} $$(5)F−<!-- - -->measure=2×<!-- ? -->Precision×<!-- ? -->RecallPrecision+Recall.$$\begin{array}{} \displaystyle F-{\rm measure} =\frac{\rm 2\times Precision\times Recall}{\rm Precision+Recall}. \end{array} $$(6)

(ii) Results analysis: Figures 2 and 3 show the extensibility of pattern changes over the process of iteration in different ways under different seed-selection strategies.

Figure 2

Figure 3

The recall rate exhibits a noticeable tendency in the following way: within a certain range, as the number of iterations increases, the number of valid patterns increases and the recall rate rises accordingly. As the number of iterations exceeds a certain range, however, the number of valid patterns extracted by the bootstrapping method gradually stabilizes, so does the recall rate. In terms of precision, an evident distinction between COM-SEED and GEN-SEED is recognized and can be described as follows: under the former strategy, the precision is capable of maintaining a high level, whereas under the latter strategy, a clear decreasing trend appears after a temporary stability in the early iteration period. The preceding phenomenon is consistent with our conclusion made in the previous section, in which the integration of the two types of words as initial seed words (i.e. COM-SEED) will refine the contexts and make them better accordance with the extraction range.

As shown in Figure 4, for the “Predicate + Object” type of pattern, the difference between the two seed-selection strategies in the performance on recall is insignificant: the precision rate under GEN-SEED exhibits a sudden decrease after a certain number of iterations in which COM-SEED remains stable.

Figure 4

As seen in Figure 5, regarding the “Subject + Predicate” type of pattern, the GEN-SEED strategy possesses advantages over the COM-SEED strategy both in precision and recall, although it is slightly inadequate in the stability of precision. It can thus be observed that the COM-SEED strategy is more suitable for extracting the “Predicate + Object” type of pattern, where the GEN-SEED strategy is more suitable for extracting the “Subject + Predicate” type of pattern. Figure 6 presents the performance of the extensibility of patterns in a special circumstance, in which only the COM-SEED strategy is executed to extract patterns in the form of “Predicate + Object,” and meanwhile only the GEN-SEED strategy for “Subject + Predicate.” From Figure 6, on the experiments of within-field extensibility of patterns, the performance of our method on precision reaches a sufficiently high level, and an acceptable level of recall can be guaranteed at the same time.

Figure 5

Figure 6