A New Citation Recommendation Strategy Based on Term Functions in Related Studies Section

Local citation recommendation aims to recommend citations for a specific context where citations are needed, which is also called context-aware citation recommendation (Tang, Wan, & Zhang, 2014). Here, the specific context can be one sentence or several sentences; in this paper, it represents all the sentences in a paragraph in the related work sections. Since contexts contain rich semantic information, content-based approaches were usually used in LCR.

He et al. (2010) proposed an effective context-aware citation recommendation approach, which designed a non-parametric probabilistic model to measure the relevance between a citation context and a document. In a similar task, the translation model was used to translate the query terms in the citation context, which bridged the vocabulary gap between the citation context and recommended document (Lu et al., 2011; He et al., 2012). Rokach et al. (2013) presented a supervised learning method utilizing three types of features (general features, author-aware features, and context-aware features) to recommend citations for a given citation context. This approach had been applied for citation recommendation service in CiteSeerX digital library. Except for these features, time or publication date has also been proven to be important in citation recommendation (Gao, 2016; Jiang, 2015; Jiang, Liu, & Gao, 2014; 2015). To provide a personalized citation recommendation service, Liu, Yan, and Yan (2013) combined user profiles with context-aware citation recommendation. Experimental results showed that the proposed strategy outperformed language model-based and translation model-based algorithms. Other research explored the effectiveness of citation networks (Jiang et al., 2018; Livne et al., 2014; Son and Kim, 2018), core scientific concepts (Duma et al., 2016) in local citation recommendation. (Duma & Klein, 2014) introduced a citation resolution method to evaluate context-based citation recommendation systems. Citation context, as the primary evidence to recommend citations, has become a theorem in this field (Bhagavatula et al., 2018).

Recently, with the popularity of deep learning in both academia and industry, different deep learning algorithms have been applied to citation recommendation. (Huang et al., 2014) represented words and documents by learning simultaneously from citation context and cited document pairs; the probabilistic neural model was used to estimate the probability of words appeared in the citation context under a candidate reference paper. This approach achieved 5% improvement on Recall than the translation model. Ebesu and Fang (2017) proposed a neural citation network (NCN) that can model the semantic composition of citation contexts and corresponding cited documents titles by exploiting author relations. This method improved several state-of-the-art baselines on all metrics (Recall, MAP, MRR, and NDCG) by 13–16%.

Unlike LCR, global citation recommendation targets on recommending a reference list for a given paper (Tang, Wan, & Zhang, 2014). However, only focus on finding the relevant papers rather than the essential papers can be a drawback of GCR because users need to evaluate the quality of the recommended papers (Chen et al. 2011). This is challenge for new researchers.

To bridge this gap, Küçüktunç et al. (2012; 2013; 2015) recommended a diverse reference list which allowed the users to reach either old, well-cited, well-known research papers or recent, less known ones. To improve users’ experience, Wu et al. (2012) proposed to recommend references based on their information needs, such as publication time preference, self-citation preference, co-citation preference, and publication reputation preference. However, most existing research used a single data source for the recommendation, which failed to meet users’ information needs about different aspects in writing (Zarrinkalam & Kahani, 2012). Therefore, Zarrinkalam and Kahani (2012) introduced a strategy to recommend citations based on multiple linked data sources, which enriched the background data of recommender systems thus enhancing the recommendations.

Kates-harbeck and Haggblade (2013) recommended references for a given abstract with a set of key technical words, an author list, and publication data using a two-stage methodology. In the first stage, they trained a classifier to rank a candidate reference list based on the paper score; in the second stage, they re-ranked the scored candidate papers using connectivity information. Their experiment results showed that text-based features were most effective in rankings. Caragea et al. (2013) compared the performance of the collaborative filtering-based (CF) approach and singular value decomposition-based (SVD) approach on GCR using CiteSeer dataset, finding that SVD performed better than CF since SVD can easily incorporate additional information.

As a benefit of different citation recommendation tasks and algorithms mentioned above, many citation recommendation systems have been developed, for example, ActiveCite (Zhou, 2010), CiteSight (Livne et al., 2014), RefSeer (Huang et al., 2014), ClusCite (Ren et al., 2014), DiSCern (Chakraborty et al., 2015), and Rec4LRW (Sesagiri Raamkumar, Foo, & Pang, 2015). Due to the usefulness but challenge of this task, citation recommendation is attracting more and more attention from researchers.

Term function means the semantic role that a term plays in the scientific literature (Cheng, 2015). The term's function also reflects the function of the citation context it locates in the scientific literature. For example, support vector machine (SVM) is the core research problem in (Li, Wang, & Wang, 2010) but the method to solve image classification problems in (Melgani & Bruzzone, 2004). Therefore, the paper “A Tutorial on Support Vector Machines for Pattern Recognition” (Burges, 1998) is cited as a related research problem in (Li, Wang, & Wang, 2010) but cited as a related method to solve image classification problems in (Melgani & Bruzzone, 2004). In other words, citations could be recommended according to the “term function” needed in the citing article, supposing the related literature in the citing is organized by these term functions. To verify this hypothesis, we develop a term function classification scheme by combining categories from existing literature and three new categories, which we draw from a pilot study, as shown in Table 2. The reason that we add three new functions is existing categories of term functions are proposed for sentence-level. Paragraphs have more complex structures than sentences, and they are usually organized by different roles in the related studies section. For example, a paragraph discusses how the method used in the original paper has been applied to solve other problems; we define this pattern as “Method+ Problem.” On the other hand, a paragraph may also focus on discussing how other relevant methods have solved the research problem mentioned in the original paper except for the method proposed in the original paper; we define this pattern as “Problem+ Method.” We also define another function, “Topic-irrelevant,” since some paragraphs that served as background knowledge are not closely relevant to the original paper's topics. Fig. 3 demonstrates that the three new term functions frequently appear in the related studies section.

We collected a dataset consisting of 238, 109, and 184 research papers in sentiment analysis, information extraction, and recommender systems, respectively. We manually downloaded the pdf files from ACL Anthology and placed them in three separate folders. We converted the pdf files into text format using Apache PDFBox

https://pdfbox.apache.org/

Paragraph organization pattern annotation is difficult since it requires extensive domain knowledge. To ensure the data quality, we invited two Ph.D. students who are familiar with sentiment analysis, information extraction, and recommender systems to annotate the data. Before annotation, they were required to pre-annotate ten articles under each folder to understand the annotation guideline, shown as follows:

Firstly, read the title and abstract to get the research problems and research methods of the article.

Fig. 2 describes an annotation example, which was saved into the text format. The pairwise Kappa coefficients (Viera & Garrett, 2005) were applied to calculate the inter-annotator agreement. Cohen's kappa scores were 0.778, 0.871, and 0.748 on sentiment analysis, information extraction, and recommender systems, respectively. These are satisfactory according to the existing evaluation standard (Viera & Garrett, 2005).

Figure 2

To investigate how authors organized literature in related work sections, we conducted a statistical analysis of the annotation results. Fig. 3 represents the distribution of term functions under each category over different topics. Based on the results, most paragraphs belong to problem+ method, problem, method+ problem, and method, with a percentage of 38.2%, 29.5%, 15.6%, and 6.0% on average, respectively, but with a slight difference between different topics. The result supports the hypothesis that authors usually pay more attention to the research problems and methods relevant to their topics while writing a literature review. In other words, they tend to focus on investigating who was involved in studying the problems and methods, how they described the problems and methods, what methods have been used to solve the problem, and how the method was applied to other problems. Existing datasets, evaluation methods, tools, and applications related to their research were also occasionally involved in two or more independent paragraphs. In addition, some topic-irrelevant materials were referred to in a few paragraphs, suggesting that the research topic is novel, and existing literature in this area was lacking.

Figure 3

The findings enlighten us to conduct citation recommendation in the related work sections by considering these information needs. In this paper, we explored this innovative citation recommendation task based on four major term functions: problem+ method, problem, method+ problem, and method.

Definition 1 (Original document collection). Given a document d with t and a as its title and abstract, the document has a set of paragraphs P = {p₁, p₂, …, p_i} in the related works section, the term function of each paragraph p_i is defined as l_i. Therefore, the paragraph set P has a corresponding term function set L = {l₁, l₂, …, l_i}. All the original documents together with their paragraph sets and term function sets form the original document collection D.

Definition 2 (Candidate document collection). Given a document c with t and a as its title and abstract, we define its term function as F = {f₁, f₂, f₃, f₄}, where f₁, f₂, f₃, f₄ refer to problem+ method, problem, method+ problem, and method respectively, which are determined by the term function of its citing paragraphs. All the candidate documents were extracted from the references of the original documents; they form the candidate document collection C.

Definition 3 (Term function-based citation recommendation). Given a paragraph p whose topic is t and expected term function is f, our goal is to estimate the probability of each document in the candidate document collection C to be cited in this paragraph.

In this paper, we explore the influence of term functions in citation recommendation and propose the term function weighting-based recommendation model. Compared with the original BM25 model and Word2vec-based VSM, an essential step of term function weighting-based recommendation models is to compute the weight of each document in the candidate document collection on each of the four term functions.

We define F = (f₁, f₂, f₃, f₄) as the weighted term function of each document d in the candidate document collection, where f₁, f₂, f₃, f₄ respectively represents the frequency of problem, method, problem+ method, method+ problem. After that, the term function should be added to the three recommendation models to improve performance.

For a document d and a paragraph as a query q, q_f donates the expected term function of this query/paragraph. Therefore, we calculate the relevance score on the term function dimension as follow: (3) p(qf,d)=ff1+f2+f3+f4,f∈{f1,f2,f3,f4} p\left( {{q_f},d} \right) = {f \over {{f_1} + {f_2} + {f_3} + {f_4}}},f \in \left\{ {{f_1},{f_2},{f_3},{f_4}} \right\}

For example, if a user inputs a query q with the term function as problem, one of a document d in the candidate document collection whose term function is F = (4, 2, 8, 1), we compute its term function weight on problem, method, problem+ method, method+problem respectively as 415 {4 \over {15}} , 215 {2 \over {15}} , 815 {8 \over {15}} , 115 {1 \over {15}} , where the sum of the weights should adhere to the following constraint: (4) ∑j=14weightj=1(weightj∈[0,1]) \sum\nolimits_{j = 1}^4 {weigh{t_j} = 1\left( {weigh{t_j} \in \left[ {0,1} \right]} \right)}

Finally, the relevance score of a candidate document d with respect to a paragraph as a query q which combines text similarity and term function matching is computed with the following two equations respectively: (5) Score(q,d)=(1+p(qf,d))⋅∑inlogN−n(qi)+0.5n(qi)+0.5⋅fi⋅(k1+1)fi+k1⋅(1−b+b⋅dlavgdl) Score\left( {q,d} \right) = \left( {1 + p\left( {{q_f},d} \right)} \right) \cdot \sum\nolimits_i^n {log {{N - n\left( {{q_i}} \right) + 0.5} \over {n\left( {{q_i}} \right) + 0.5}}} \cdot {{{f_i} \cdot \left( {{k_1} + 1} \right)} \over {{f_i} + {k_1} \cdot \left( {1 - b + b \cdot {{dl} \over {avgdl}}} \right)}} (6) Score(q,d)=(1+p(qf,d))⋅q⇀*d⇀‖q⇀‖‖d⇀‖ Score\left( {q,d} \right) = \left( {1 + p\left( {{q_f},d} \right)} \right) \cdot {{\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} *\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \over {\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} } \right\|\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \right\|}} Equation (5) is for term function weighting-based BM25 model, while equation (6) is for Word2vec-based VSM models with term function weighting.

Pseudo code for the baseline recommendation models or term function weighting-based recommendation models is described in Algorithm 1.