An explorative study on document type assignment of review articles in Web of Science, Scopus and journals’ websites

As Garfield (1996) described,

“Reviews play an essential role in scientific communication and understanding. In terms of the inherent characteristics of the review, they can provide a synthesis of the proliferating fragmented knowledge appearing in the plethora of foreign and domestic journals in a specialty or subspecialty. As such, they can elucidate trends in research and point to unanswered questions that provide opportunities for future study. Reviews also give science policymakers as well as researchers a clearer insight into the potential importance of emerging knowledge.”

In addition, reviews provide excellent and stimulating reading for the general reader and researcher dedicated to cross-disciplinary study, because they advance our perceptions of the relationships between different research efforts. The value of a review does not exist solely in the author’s synthesis of previously published papers; the bibliography in a review usually is a high quality list of core articles about the subject. In all, writing a review will certainly do as much for the advancement of science as those who do the original research.

Despite the importance of reviews in science and science communication, the effect of reviews on scientometrics analysis is also significant. Reviews tend to be more frequently cited (Aksnes, 2003; Moed, 2010; Teixeira et al., 2013). Correlated with this overcitation, there is an overrepresentation of reviews in the highly cited papers, and this overrepresentation becomes greater when the most highly cited papers are considered (Miranda et al., 2018). Moreover, 20% reviews can increase the average citations of an individual researcher by 40%–80%. Consequently, researchers boost their citations by publishing reviews, and journals increase their Impact Factor by publishing reviews (Ketcham et al., 2007; Lei et al., 2020; Teixeira et al., 2013).

Review a will also affect the citation of the articles reviewed. An alarming trend within the biological/biomedical science has been noted that the authors prefer to cite review articles rather than the original article when writing literature review(Ketcham et al., 2007; Teixeira et al., 2013). It is more efficient to cite reviews than all the individual studies, but the scientific credit to the time-consuming original studies will be absorbed by the reviews and the review authors (Ketcham et al., 2007; Lachance et al., 2014; Teixeira et al., 2013). Ho et al. (2017) out pointed that review papers will affect the main path analysis and clustering analysis. When conducting bibliometrics research and evaluating scientific research achievements, we should decide which document type to be included and whether to treat articles of different document types separately(Lei et al., 2020). To facilitate the above process, highly accurate assignment of review articles in databases is required. Wrongly assigned document type has great impact on the citation-based evaluation(Donner, 2017; Zhu et al., 2022).

WoS describes the Review^① as

“Detailed, critical surveys of published research. A review article may summarize previously published studies and draw some conclusions but will not present new information on the subject. Includes Reviews, Review of Literature, Mini-reviews, and Systematic reviews. If an article is listed under the review section in a journal and/or Review of Literature appears in the title it will be assigned a review.

If an article is not assigned a review by the journal but Review, Systematic Review or Mini-review appears in the title, it must also appear someplace else in the article (abstract/summary or introduction) in order to be assigned the document type review.

NOTE: If the article(s) meet the above criteria - they must have References in order to be tagged as a Review item.

Review articles that were presented at a Symposium or Conference will be processed as Proceedings Papers.”

This description has been accessed recently. Several years before, the description of “Review” in WoS is a renewed study or survey of previously published literature providing new analysis or summarization of the research topic(WoS, 2023). And several criteria are used to determine whether a paper is a review. such as the following:

“In the JCR system any article containing more than 100 references is coded as a review. Articles in ‘Review’ sections of research or clinical journals are also coded as reviews, as are articles whose titles contain the word ‘Review’ or ‘overview’”

The “more than 100 references” criteria had been removed in 2010. The change of criteria may lead to some changes to the statistics. For example, in a 1987 paper, Eugene Garfield pointed out that there were 625,432 articles indexed in the 1986 SCI and of which approximately 32,000 are reviews (Garfield, 1987). But when we search the SCI in June 2021, total articles indexed in 1986 are 709,136 and 13,197 are reviews.

Scopus describes Review as

“A significant review of original research also includes conference papers. Reviews typically have an extensive bibliography. Educational items that review specific issues within the literature are also considered to be reviews. As non-original articles, reviews lack the most typical sections of original articles such as materials & methods and results”

In Scopus, there is another document type related to review called “Short survey”, which is described as

“Short or mini-review of original research. Short surveys are similar to reviews, but usually are shorter (not more than a few pages) and with a less extensive bibliography.”

We can see that the description of reviews in Scopus and WoS is mainly related to the words used in titles and abstracts, the length of the reference list and article structure. Colebunders et al. (2013) compared the number of records related to reviews retrieved in WoS via different strategies, i.e., (1) based on the WoS document type, (2) having either the word review or the word overview in the title, and (3) a topic search (TS=) for the words review or reviews. It is found that the absolute and relative numbers of reviews differ depending on which of the three definitions are used. Harzing (2013) reported a comprehensive analysis of document categories for 27 journals in nine Social Science and Science disciplines and showed that WoS may misclassify social science journal articles containing original research into the “review” or “proceedings paper” category. The possible reason is length of references of in social science articles is larger than 100.

Several studies compared the document type assignment accuracy of citation index database with other sources(e.g., manually coded, publisher’s websites). Hayashi et al. (2013) compared the records’ document type of 18 research journals of Nature Publishing Group in WoS, Scopus, and the website and found that all “Review” items in the website were labeled as “Review” in both WoS and Scopus, and some papers of other types are also labeled as “Review” in WoS and Scopus. As the authors didn’t further report the details of these reviews labeled in WoS and Scopus, we cannot infer the real accuracy. Donner (2017) reported a study on the document type assignment accuracy of 791 randomly selected papers in WoS and Scopus. When only focusing on these selected papers, the accuracy of WoS (83%) is higher than Scopus(76%). The study also statistically inferred the WoS overall proportion of correctly assigned DT is 0.94, but for the reviews, the precision is 0.87 and recall is 0.57. Yeung (2019) examined the DT assignment accuracy of 400 top-cited publications defined by Scopus as ‘article’ in the field of food and nutritional sciences. Among these 400 publications, 117 were manually coded as reviews. Further, for these 117 reviews, 111 is indexed in WoS and 55/111 were wrongly labeled. Another interesting observation is that, the publisher' website labeled 52/117 reviews as articles.

Reviews have always been a very important research object in the field of Scientometrics and Informetrics, and the research directions about reviews had been discussed at a workshop at the Conference of the International Society of Informetrics in September 2019, in which participants identified six realms of study. One of the themes is “the study of methodological caveats resulting from the usage of scholarly databases”, such as the lack of accuracy of document assignation in scholarly databases (Blümel et al., 2020). In this work, we’d like to analyze the accuracy of document type assignment of review articles in WoS and Scopus on a large scale and identify the possible reasons for wrongly assigning.

In the publishing ecosystem, there are several journal series mainly (or only) publishing review articles, e.g., the Annual Reviews series, Nature Reviews series. These journals can be treated as appropriate data sources for us to investigate the correctness of document type assignment of reviews in databases. For example, as shown in Figure 1, a paper published by Nature Reviews Cancer, has a document type annotation on its official website, and can be further compared with the corresponding document type provided in WoS and Scopus. In this study, we selected 160 SCI journals included in Journal Citation Report 2019 from five series of pure review journals (only publishing review articles) and four series of mixed review journals (mainly publishing review articles) as shown in Table 1.

Figure 1.

As JCR 2019 covering papers published during 2017-2018, the website annotation information of papers published in the above review journals during 2017-2018 is collected from the journals’ official websites as the basic dataset. We collected 27,616 papers, and for these collected papers, we retrieved their document type information from WoS and Scopus using Digital Object Identifier (DOI). Because of the problems such as record missing, errors or duplicate DOI in WoS and Scopus, we use the paper title, journal name and other supplementary information to manually match unmatched records.

For pure review journals and mixed review journals, the document type of each paper is assigned based on the journal section headings or the document type annotation on the paper’s official website, see the examples shown in Figure 2. As we mainly focus on the assignment of reviews and different journals and databases have different names for the same document type, to facilitate the analysis, we grouped these document type names into “reviews”, “articles” and “other papers”(e.g., editorial material, correction). We further divide the reviews/articles into explicit review/article (the section heading or the website annotation directly indicates its document type) and implicit review/article (the section heading or the website annotation doesn’t directly indicate its document type). The details of the division are described for each journal series in the Results section. “Short survey” is a special document type in Scopus, and we keep its original name.

Figure 2.

To measure the assignment accuracy of WoS and Scopus compared against official websites, we construct an assignment matrix and calculate the corresponding precision, recall, and F₁-score metrics(Baeza-Yates et al., 1999; Davis et al., 2006) as follows, Precision=Ndbweb/NdbRecall=Ndbweb/NwebF1−Score=2/(1/Precision+1/Recall)$$\matrix{ {{\rm{Precision}} = N_{{\rm{db}}}^{{\rm{web}}}/{N_{{\rm{db}}}}} \cr {{\rm{Recall}} = N_{{\rm{db}}}^{{\rm{web}}}/{N^{{\rm{web}}}}} \cr {{{\rm{F}}_1} - {\rm{Score}} = 2/(1/{\rm{Precision}} + 1/{\rm{Recall}})} \cr } $$ where N^web_db is the number of papers marked as review both in the website and database, N_db is the number of papers marked as review in the database, and N^web is the number of papers marked as review on the website.

In the above sections, we illustrated the comparison of document type assignment across websites, WoS and Scopus for each review journal series. Here we summarize the assignment performance of WoS and Scopus as shown in Figure 8.

Figure 8.

Figure 8(a) and Figure 8(b) respectively represent the results of total precision (including explicit review and implicit review) and total recall for each journal series. In general, WoS and Scopus have high performance toward total precision(exceeding 97%) in most journal series. However, the total precision of RMP is 0% in WoS and Scopus, and the precision of Nature Reviews series is 78.78% in Scopus. WoS and Scopus show some differences in terms of recall. For example, Scopus performs better for Cell Trends In series and Current Opinion series, while WoS performs better for Annual Reviews and Taylor & Francis Critical Reviews. When the recall of explicit reviews and implicit reviews are displayed separately, we can see a huge difference. Compared with the recall of explicit reviews shown in Figure 8(c), the recall of implicit reviews shown in Figure 8(d) is much smaller, and the performance of WoS is better than Scopus. This observation implies that the two databases should pay more attention to these implicit reviews in document type assignment, and publishers can better assign the labels on their websites.