Lone Geniuses or One among Many? An Explorative Study of Contemporary Highly Cited Researchers

The drive for research excellence has become ever more pervasive since the turn of the millennium. Funders, policymakers, stakeholders, research institutions, journals, and individuals all strive for excellence as the holy grail of academic life (Lamont, 2009; Moore et al., 2017; van Leeuwen et al., 2003). As the research policies of numerous countries increasingly emphasise excellence, they are consequently developing evaluation systems to identify universities, research groups, and researchers that can be said to be “excellent” (Danell, 2011). However, in this process, the notion of excellence has become a “contested concept”. While praised by many, others now see it as an empty term that may do more harm than good for the conduct and impact of research. By rewarding certain approaches, themes, collaboration patterns, and types of researchers at the expense of others, it is argued—for example—that it reinforces existing hierarchies, undervalues local and contextualized knowledge, and limits diversity (Ferretti et al., 2018; Gallie, 1955; Moore et al., 2017; Stilgoe, 2014; Vazire, 2017).

One of the main challenges is that excellence as broadly understood is produced and defined in a multitude of sites and by an array of social actors and, therefore, is difficult to capture in a standardized and consistent manner (Lamont, 2009). It may look different across different fields of research, between different review contexts, and between various national policy contexts (Langfeldt et al., 2020). Therefore, the discomfort with the concept of excellence is particularly pronounced when it becomes standardized and whenever proposals are made to measure it. This dilemma is well known in numerous contexts: simplification is an inescapable avenue in any attempt to represent complex concepts with numbers.

In this context, citation indicators have been brought forward and increasingly used as a potential means to quantify and measure research excellence. Since the turn of the millennium, the bibliometric community has been examining indicators that reflect the top of the citation distribution, such as the number of “highly cited” or “top” articles (Aksnes, 2003; Bornmann, 2014; Tijssen, Visser, & van Leeuwen, 2002; van Leeuwen et al., 2003). Such metrics are often explicitly linked to notions of scientific excellence and to the assumption that highly cited papers represent scientific breakthroughs or research of particular importance. In science policy contexts, they are frequently called upon to legitimate policy interventions, funding and publishing choices, and hiring and promotion decisions (Ferretti et al., 2018; Wilsdon, 2015).

However, the interest in highly cited publications and researchers is not new. Originally, such analyses gained prominence through the work of Eugene Garfield, who produced lists of most-cited researchers over the years. In particular, Garfield had an interest in using citation data to forecast Nobel Prize winners by identifying a group of researchers he termed “of Nobel class” (Garfield & Welljams-Dorof, 1992).

Lately, new versions of such rankings and lists of highly cited scientists have received much public attention, exemplified through, for example, Clarivate Analytics’ Highly Cited Researchers list (Web of Science Group, 2018) and Nature's Rising Star Index (Dequilettes et al., 2018), although the latter index has the main focus on institutions (www.natureindex.com). The Highly Cited Researchers list from Clarivate Analytics can be considered an extension of Garfield's work in recognizing investigators whose citation records position them in the top strata of influence and impact. Underlying these lists lies a perception of the highly cited scientists as individual geniuses who can be singled out for their extraordinary contributions. As such, these lists often contain strong value-laden statements and interpretations. For example, it is stated, “The 2018 Highly Cited Researchers from Clarivate Analytics is a contribution to the identification of that small fraction of the researcher population that contributes disproportionately to extending the frontier and gaining for society knowledge and innovations that make the world healthier, richer, sustainable, and more secure” (Web of Science Group, 2018).

However, the question is how valid this perception is in the age of globalization, research collaboration, and team science (Wagner, 2008). What types of research and what types of researchers do such lists actually capture and do they correspond to the underlying understanding of highly cited researchers (HCRs) as lone geniuses who single-handedly extend the frontier of our knowledge and make the world a better place? In order to assess this question, an investigation was conducted of 150 HCRs from five different countries (Norway, Denmark, Sweden, the Netherlands, and the UK) who appear on the Clarivate Analytics’ Highly Cited Researchers list (2018). From this outset, the present study examines the following key questions: a)

What characterises the scientific performance of the HCRs in terms of productivity and citation rates? Here, we analyze their highly cited papers as well as their total publication output. More specifically, we assess the extent to which the HCRs are mass producers of publications or whether they have produced a more limited number of publications during the time period analysed.

By diving into these questions, this study helps to develop further knowledge on HCRs. Who becomes highly cited and what type of research benefits by defining excellence in terms of citation scores and specific counting methods? Based on these investigations, the study contributes to the ongoing critical discussions concerning the concept of excellence measured from a bibliometric perspective. Hereby, we highlight the tensions between the common perceptions of scientific excellence (e.g. focus on major scientific breakthroughs and discoveries) and research becoming highly cited involving a large number of authors.

The article proceeds in the following manner: Section 1.2 presents a short review of the literature of highly cited researchers and publications. Section 2 presents our methods and data. Section 3 outlines the results, and section 4 contains the discussion and conclusion.

Citation distributions are very skewed. Most publications obtain none or only a few citations, while a small proportion of the publications become very highly cited (Aksnes, Langfeldt, & Wouters, 2019; Seglen, 1992). This phenomenon has been well known since bibliometrics began flourishing as a research field over half a century ago (Price, 1965). Naturally, the skewed distribution has drawn a lot of attention—for example, relating to the causes that may explain it (Aksnes, 2003), what implications it has for science policy and for the use of citations as performance indicators (van Leeuwen et al., 2003), as well as for the statistical properties of citation indicators (Schmoch, 2020).

Highly cited papers are usually considered as contributions with particularly substantial scientific influence or impact. Various sorts of justifications for this association have been provided. The most basic relates to the reference practice of scientists (Aksnes et al., 2019). When writing a paper, researchers refer to prior studies that have been relevant or useful for their own research. Papers that have been highly cited have accordingly been useful for many more preceding studies than articles that are barely cited or not at all. This argument dates back to Robert K. Merton's norms of science, according to which scientists are obliged to cite the work they rely on and credit contributions by others (Merton, 1979). However, numerous studies have shown that the referencing process is also influenced by a multitude of other factors (Bornmann & Daniel, 2008), which implies that the association between high citation counts and scientific impact is complex. This is particularly the case when citation counts are interpreted as measures of second-order concepts, like scientific importance or quality. Quality is a multidimensional concept, consisting of different dimensions such as solidity, plausibility, originality, and scientific value. According to a review by Aksnes et al. (2019), there is little evidence that citations reflect these latter dimensions, while scientific impact may be considered a more appropriate interpretational term. Second, justifications have been provided by comparing citation metrics with peer judgements of scientific quality. Over the years, a large number of such studies have been conducted (for an overview see e.g. Aksnes et al., 2019; Wouters et al., 2015). These have shown that the relationship is not unambiguous and the correspondence reported has been moderate in most studies. This implies that the empirical support for claiming that citations reflect the same aspects of scientific quality as peer review judgements is limited.

A few comparative studies have specifically addressed the issue for highly cited papers. For example, in a survey among authors of highly cited Dutch papers, Tijssen, Visser, and van Leeuwen (2002) found that the authors’ ratings of these papers were mixed. Although for the large majority, there was a strong positive correspondence between the perceived scientific quality and the citation counts, a minority (15%) did not perceive the quality of their papers to be of international “world class” level. In a similar Norwegian study by Aksnes (2006), the majority (74%) of the highly cited papers were considered to represent major contributions by the authors themselves, while the remainder were rated as intermediate or minor contributions. Further, Porter et al. (1988) reported that only about one-third of the articles nominated by authors as their best were also their most-cited publications. These findings indicate that highly cited articles do not necessarily represent major scientific achievements, at least not according to the authors themselves. Assumedly, the correspondence would have been even weaker if using stronger superlatives, such as “breakthrough science” or “cutting-edge research” for characterising these papers. Nevertheless, it is argued by Tijssen, Visser, and van Leeuwen (2002) that highly cited articles can be used as a valid measure of academic scientific excellence, but only at aggregated publication levels.

As noted above, the interest in highly cited publications and HCRs goes back to Eugene Garfield, who—over a long period of time—investigated this issue in a Nobel prize context (Garfield, 1986; 1992). He found that almost all Nobel laureates were very highly cited within their fields by publishing highly cited articles or citation classics. Findings like these may also be considered as a kind of validation, as there is apparently a strong correlation between high citation counts and achievements of the world's most prestigious scientific award. However, Garfield also identified numerous HCRs who did not obtain the Nobel prize—for example, the authors of the most cited article ever (Lowry et al., 1951). Therefore, one may turn the question around and ask the following question: Have all HCRs contributed to breakthrough science? According to Garfield, the answer to this question is no, as “[…] citation frequency by itself is not adequately indicative of outstanding and influential publication” (Garfield, 1986) and similarly, “[…] it would be absurd to claim that a researcher deserves the Nobel Prize simply because he or she is a citation superstar”. (Garfield, 1992).

Overall, it appears reasonable to conclude that there remains limited support to the claim that indicators of highly cited papers and researchers reflect scientific excellence. Nor is it clear how they relate to various aspects or conceptualizations of excellence. Such indicators have nevertheless increasingly been used in the context of research evaluation. Already in 2001, the European Commission applied highly cited papers as an indicator when comparing the research performance of countries in the European Union (EU) (European Commission, 2001) and there were scattered examples of such applications even earlier (Martin & Irvine, 1983; Plomp, 1994). Today, such indicators are used in a variety of contexts, including university rankings, such as the Leiden and Shanghai rankings (the Academic Ranking of World Universities)

See: https://www.leidenranking.com/information/indicators and http://www.shanghairanking.com/ARWU-Methodology-2019.html

Generally, the HCRs are extremely productive. On average, each individual published 151 articles during the period 2006–2016, while the median researcher published 121 articles. The range of the production varies from a minimum value of 14 to a maximum value of 730 articles, as depicted in Figure 1.

Overall, the HCRs have published 2,259 highly cited papers (top one percentile), which corresponds to an average of 17 highly cited papers per person. This implies that, on average, 16.5% of the publication output of each researcher is within the top percentile. This is almost 17 times higher than the “expected” value of 1%. Nevertheless, the large majority of the publications of the HCRs do not reach the highly cited threshold. The proportion of highly cited papers varies significantly across the individuals, cf. Figure 1. In the most extreme case, 69% of the publication output is within the top percentile. However, in general, the proportion of highly cited papers reduces as the total number of papers per HCR increases.

Figure 1

It must be noted that there is a certain overlap in the sample, as 351 HC-papers have been authored by more than one of the individuals analysed. Accordingly, the number of unique highly cited papers is 2,060, which implies that 17% of the papers are attributed to more than one HCR. This indicates that some of the HCRs belong to the same research groups/networks.

Figure 2 provides further information on the distribution of HCRs according to the number of highly cited papers. While the average is 17 highly cited articles during the period, 6% of the researchers have five or fewer such articles and 14% have more than 25. Thus, for simplification, the researchers may be divided into two different groups: Researchers who do not regularly contribute to high-impact research (i.e. having only a few papers with high impact) and researchers who steadily contribute to high-impact research.

Figure 2

The annual publication output of each HCR was analyzed separately. The results reveal that almost all researchers have published at least one paper each year (Figure 3). The majority of the researchers have also published at least one highly cited paper annually—the proportions vary from 53% to 79%. Thus, many researchers have published highly cited papers over longer periods.

Figure 3

As described above, contributions in terms of review papers and guidelines were identified. A total of 15% of the highly cited papers were review articles. Thus, this document type appears rather frequently in the set of highly cited papers. At the level of individuals, 7% of the HCRs had mainly or entirely published highly cited review papers. Thus, their presence on the list of HCRs is due to the publication of such papers, most of them having a large number of additional authors.

Guidelines, of which the large majority represents clinical guidelines, also appeared rather frequently and accounted for 6% of the highly cited articles. This proportion would have been significantly higher if only papers in the field of medicine had been included as a denominator in the calculation. When analyzing the highly cited papers of the researchers, 5% of the individuals had mainly or entirely published guidelines.

The large occurrence of guidelines is also evident when analyzing the frequency of title words in the sample of highly cited papers. Here, “guidelines” is one of the most frequently appearing title words, along with words such as “meta-analysis” and “review”, which probably represent words primarily relating to review papers.

Not unexpectedly, we find that many of the highly cited papers have been published in high-impact journals. Table 2 provides an overview of the journals that account for the largest number of articles. On the top of the list, we find Nature and Science; together with other very prestigious journals on the list, such as Proceedings of The National Academy of Sciences of the USA (PNAS), New England Journal of Medicine, and the Lancet. However, it must be noted that not all articles have been published in high-impact journals, as illustrated by the presence of PLoS One. This is one of the largest journals in the world (Brainard, 2019). Table 2 also presents similar statistics for the other papers of the HCRs. This journal list looks rather different and mainly comprises field-specific journals.

Next, the highly cited articles of the HCRs have been analyzed using data on the number of co-authors. Generally, the highly cited papers have a very large number of authors, on average 59, and in the most extreme case over 500. The latter group consists of 1.7% of the articles.

For each individual, we calculated the average number of authors of their highly cited papers. The distribution is illustrated in Figure 4. There are large variations in the distribution. Overall, 15% of the HCRs published highly cited papers with an average of over 100 authors, and 7% with 50–100 authors. Thus, these scientists tend to be members of large research consortia and their presence on the list can be attributed to such memberships. There is also a significant share of researchers who publish with very large research groups—20–50 authors (20% of the individuals). However, there are also researchers who have fewer co-authors on average. The highly cited papers of 9% of the HCRs have five or fewer authors.

Figure 4

Figure 5 depicts a scatter plot where the average number of authors of the HC papers is compared with the average number of authors for the other papers of the HCRs. The trend line shows the correlation between the datapoints and is based on linear least squares regressions. Generally, there is a rather strong positive linear correlation (R² = 0.4), which implies that the collaboration profile is similar: Authors publishing highly cited papers with many co-authors also tend to publish other papers with many co-authors. Nevertheless, there are certain individuals who deviate from this pattern. The gradient or slope of the trendline is 3.6.

Figure 5

Figure 6 depicts the difference in the average number of authors of the highly cited papers and the other papers by individuals. For approximately one-third of the HCRs, there is a difference of over 15 authors. For this group, the highly cited research by collaboration pattern differs significantly from the other research of the individuals, thereby suggesting that this research is different—for example, by involving participation in research consortia.

Figure 6

Overall, there are 33,009 different unique author names (last name and initials) contributing to the HC-papers, although the number of “real” people is probably somewhat lower due to different variants/spellings of author names of individuals. This confirms that the highly cited research of the HCRs is very far from representing individual achievements overall. Nevertheless, the HCRs appear as first or last author in 31% of the papers, thereby suggesting that they may have had a key role in approximately one-third of the papers. This is assuming conventions for authorship where the first author is the one who has undertaken the largest portion of the research and the last author is the leader of the investigation—an assumption which is not likely to hold for all papers analysed (Waltman, 2012).

Based on the data of the institutional affiliation of the authors of highly cited papers, international collaboration profiles were analysed. A large majority of highly cited papers involve international collaboration. Nevertheless, 22% of the papers did not have such collaboration and the authors were affiliated with only one country.

The distribution of HCRs according to average number of author-affiliated countries of their highly cited papers is presented in Figure 7. A large proportion of the researchers have published such papers with co-authors from numerous different countries. In total, 13% of the HCRs have an average of over 10 different countries contributing to their highly cited papers, while 27% have 5–10 countries. The group where the highly cited papers involve international collaboration to a small extent is merely 14% (1–2 countries). In conclusion, the publications of the HCRs tend to be the results of research involving scientists from several different countries.

Figure 7

In Figure 8, the profile of highly cited papers has been compared with the profile of the other papers for each individual. In general, the highly cited papers involve multilateral collaboration to a much larger extent than the other papers. The trendline shows the correlation between the datapoints and is based on linear least squares regressions. As shown in the equation, the slope of the line is 2.9.

Figure 8

The ranking list of Clarivate Analytics is based on a so-called “whole count” methodology, where each researcher gets full credit for the papers regardless of the number of additional co-authors. This also holds for the calculation of citation indicators. Although this method remains widely used, the application of fractional contribution measures has become more common (Aksnes et al., 2019). In the simplest version, this implies that each unit of analysis would be credited with a fraction of an article based on the number of contributing authors.

Highly cited papers have been investigated in order to analyse the implications of applying these two methods. Here, the total number of citations to the highly cited paper has been counted using whole and fractionalised methods. Then, an average for each individual has been calculated. The results are illustrated in Figure 9.

Figure 9

Overall, by using the fractionalised methodology, the citation numbers of the HC-papers are reduced by 89%. This ratio varies significantly across individuals. In the most extreme case, a researcher contributing to highly cited papers receiving over 12,000 citations would be credited with only 77 citations using the fractionalised methodology. This corresponds to a ratio of 0.006. On the other hand, there are a few researchers with a corresponding ratio of 0.4. Thus, it is evident that the use of whole versus fractionalised counts has very large implications when it comes to HCRs and, in turn, also on the ranking list and identification of HCRs.