Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Classifications and rankings are based on assumptions and decisions about parameters. For example, Harvard University is listed at the top in many rankings. However, when one controls for the budget by dividing the numbers of publications and/or citations (output) by budget (input), other universities come to the fore. Leydesdorff and Wagner (2009), for example, found Eastern-European universities (Poland, Slovakia) as most efficient in terms of output/dollar, because of the relatively low costs of skilled labour in these countries at the time. The ranked order is conditioned by the choice of indicators.

In China, for example, the mathematics department of Qufu Normal University unexpectedly led the ranking of US News with a 19^th position worldwide.

https://www.usnews.com/education/best-global-universities/china/mathematics; retrieved 27 October 2020.

Comparing universities may be even more complex than comparing nations. While nations can be expected to cover portfolios of standardized measurements,

See, for example, OECD’s Frascati Manual: https://www.oecd.org/publications/frascati-manual-2015-9789264239012-en.htm

The reliability and reproducibility of yearly rankings is low because the models are based on specific choices in a parameter space of possible choices. Gingras (2016), for example, concluded “that annual rankings of universities, be they based on surveys, bibliometrics, or webometrics, have no foundation in methodology and can only be explained as marketing strategies on the part of the producers of these rankings.” Others have pointed to the reputational effects in rankings. Rankings can instantiate historical power structures (Bowman & Bastedo, 2011). However, numbers can also provide a new perspective because ongoing changes may have gone hitherto unnoticed. We shall provide examples of such unexpected developments below.

In our opinion, statisticians and bibliometricians have made progress during the last decade in developing arguments for making informed decisions about the data and relevant methodologies for comparing institutions. The rankings no longer have to be arbitrary and subjective combinations of surveys and partial indicators. However, they remain constructs which enable us to distinguish among universities in specific terms. For example, the Leiden Rankings (LR) of research universities stand out in terms of their transparency and clear objectives. The objective of the LR exercises is to rank research universities exclusively in terms of their publications and citations as output, without accounting for other differences.

We use in this study LR and not, for example, the Shanghai Jiaotong University Ranking, because the data curation is bibliometrically transparent and easily reproducible in LR. Our analytical and statistical arguments, however, can analogously be applied to the other ranking (e.g. Peters, 2017; Shin, Toutkoushian, & Teichler, 2011). In addition to providing the rankings online (https://www.leidenranking.com/ranking/2020/list), the yearly source data for LR are freely available in an Excel file for secondary analysis. This data is derived from the Web of Science of the Institute of Scientific Information (ISI/Clarivate) in Philadelphia. The address information is disambiguated and reorganized by the Centre for Science and Technology Studies (CWTS) in Leiden.

LR does not take into account conference proceedings and book publications; it is exclusively based on research articles and reviews published in the journals included in the Web of Science.

Clarivate’s criterion for classifying papers as reviews is as follows: “In the JCR system any article containing more than 100 references is coded as a review. Articles in ‘review’ sections of research or clinical journals are also coded as reviews, as are articles whose titles contain the word ‘review’ or ‘overview’” (at http://thomsonreuters.com/products_services/science/free/essays/impact_factor/ (retrieved 8 April 2012); https://clarivate.com/webofsciencegroup/essays/impact-factor/ (retrieved on 29 August 2020)).

The full counting method gives a full point to each author and her institutional address. Fractional counting divides the attribution by the number of authors or institutions. Bibliometrically, fractional counting has the advantage that each publication is eventually counted as one full point and thus percentages add up to hundred (Anderson et al., 1988). Among the possible counting rules, LR uses fractional counting at the institutional level. For example, if a publication is co-authored by three researchers and two of these researchers are affiliated with a particular university, the publication has a weight of 2 / 3 = 0.67 in the attribution of the scientific impact to this university (Sivertsen, Rousseau, & Zhang, 2019).

Another important decision is the time window used to harvest publications and citations, respectively. LR uses four-year periods for cited publications. The last interval for LR 2020 is 2015–2018; the count included citations to publications in these four years cumulatively. The resulting file with input data for this study contained ranks for 1,176 research universities in 65 countries in the preceding years in intervals of four years (Table 1). Rankings are provided both fractionally and as whole counts. In terms of disciplines, data is provided for “All sciences” and five major fields on the basis of relevant journal categories: (i) biomedical and health sciences, (ii) life and earth sciences, (iii) mathematics and computer science, (iv) physical sciences and engineering, and (v) social sciences and humanities. In this study, we limit the analysis first to “All sciences,” the last available period (2015–2018), and fractional counting. However, this analysis can be repeated analogously using subsets with other parameter choices.

Our sample contains 205 Chinese universities (Table 2) covered by LR 2020. This data was further processed by us with dedicated routines, which are online available at http://www.leydesdorff.net/software/leiden (Leydesdorff, Bornmann, & Mingers, 2019). These routines analyze the comparisons in each nation. China and the US, however, are represented in LR 2020 with 205 and 198 universities, respectively, making comparisons possible. For example, one can compare Harvard with Stanford as US universities, and—as we shall see below—it is possible to compare them with Tsinghua or Zhejiang University.

In the LR2020, universities are grouped by nation-states, but it is possible to draw samples of universities across nations or within nations. Classification of universities at lower levels of aggregation can be relevant for the study of regional innovation systems. Whereas universities remain the units of analysis in LR, relevant samples can be drawn from the database on the basis of a specific research question.

For example, if the research question is about comparing universities in the EU, one can study the EU as a single unit internally different from the USA or China. The organization at the level of EU is probably different from the sum of the national perspectives. As noted, LR also contains information for six major fields so that one can cross-tabulate nations with these disciplinary categories, where one could test in principle the conjecture that “China is strong in the basic sciences and the US in the biomedical sciences.” We will leave this for a later study, but focus here first on how to compare US and Chinese universities in a single framework.

Both the US and China happen to be represented with approximately 200 universities in LR 2020 (Table 8). Among the 198 American universities, Rockefeller University is an extreme outlier with more than 30% of the papers in the top 10% most-highly cited group. The following analysis is based on 205 Chinese and (198 – 1 (Rockefeller University) =) 197 American universities. The clustering in the network among these (197 + 205 =) 402 universities is visualized in Figure 6. Three clusters and a few isolates are distinguished by the statistical analysis. The isolates are: George Mason University and the University of Toledo in the USA, and the Hangzhou Dianzi University in China. The cross-tabulation in Table 7 is significant at the one-percent level (χ² = 93.40; p < .01).

Figure 6

Table 8 shows the top 20 Chinese universities juxtaposed to the 20 American universities with highest z-scores at different scales. This table reveals that the highest ranked among the Chinese universities (Tsinghua with z = 11.005) does not reach the z-level of the lowest among the 20 most-highly ranked American universities in the right column (z = 12.23). This may be due in part to historical factors where Chinese authors are not as well integrated into the network of science as others and thus struggle to gain citations, and it may also reflect some quality issues, as well. Figure 7 shows that the distribution of z-scores is systematically lower for Chinese universities than for their US counterparts. In other words, using these parameters, China is still far behind the US in terms of the quality of its universities.

Figure 7

The methodology for the normalization in terms of different fields of science is continuously improved by CWTS and the database is expanded with new universities. In LR 2016, for example, the number of universities covered was 842 compared to 1,076 in this study based on LR2020. The expansion of the database from year to year may have an effect on the rankings because universities may enter the comparison with higher or lower scores on the relevant parameter.

To address the problem of changes in the methodology, LR values are recalculated each year for the historical values of the indicators based on the latest methodology. Thus, we have two time series: one based on the yearly series of LR 2016 to LR2020 and one based on the reconstruction of the data using the method of LR 2020. The two series for Fudan university are graphed in Figure 8.

Figure 8

In LR 2016, Fudan University had 9.81% of its papers in the top 10% class (all fields). In 2020, the position of Fudan University has dropped to 9.03 % of its papers in the top 10% class. This is a decline of 0.78% (9.81–9.03). Would Fudan University have been able to keep the value of 9.81%, it would still have been in the second tier in 2020. Using the 2020 model, however, the value for 2016 is reconstructed as 9.54%, so that we can conclude that the difference in the data is only 0.37% (that is, 9.81–9.54%). The remaining decline (9.54–9.03 =) 0.51% is an effect of changes in the model. In other words, the model accounts for almost two-thirds of the decline (0.51/0.78 = 65.4%) and the citation data themselves for (0.27/0.78 =) 34.6%. These relative declines are of the same order of magnitude as the ones shown in Leydesdorff, Wouters, and Bornmann (2017) for Carnegie Mellon University in the USA during the period 2012–2016.

In summary, both changes in the data and changes in the model can result in differences in the rankings. Whereas scientometric indicators are meant to serve “objectivization” of the discussion about quality, the quality of the indicators themselves is also an issue in the discussion of the results which requires attention. Differences may be due to changes in the data, the models, or the modeling effects on the data.

Our main argument has been that differences among universities can be tested for their statistical significance. This allows for a classification, since universities which are not significantly different can be grouped together, regardless of geographic location. The scientometric groupings, however, should not be reified. The groupings are not stable when we use different methods. Particularly at the margins, the attribution may be sensitive to parameter choices.

The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings. Further analysis can reveal points which merit attention and discussion. Is a decline due to a parameter choice, or is there a reason for concern? An alternative view can foreground unseen relationships in the background; the resulting insights can be made the subject of managerial and political interventions. One can expect that qualitative assessments lag behind the ongoing developments. Cultural expectations are conservative, while all universities are under the pressure to change their position in relationship to one another in a competitive environment. One can zoom in and organize follow-up investigations in these cases. The result may initially be unwelcome, but can also induce asking urgent questions.

At the macro level, our results show that Chinese universities do not yet (?) operate at the same levels of performance as those in the USA. Despite concerns in the US about the “competition” from Chinese universities, the latter do not rank in the same elite categories as American universities at this time. As in other national systems, we found three or four groups of universities at national levels. In the case of China, the top-list includes five universities with an address in Hong Kong, perhaps partly because of the historical British tradition operating in the background.

University ranking services contribute to the hierarchy of reputation and status that has been the subject of critique and scholarship. Bourdieu (1998) examined the political, intellectual, bureaucratic, and economic forms of power that influence governance and the allocation of resources based upon elite reputation, claiming a link among them. This line of scholarship views international status and side-by-side comparisons, such as those found in rankings, as operating within a framework of broader geopolitical struggles (Heilbron, Boncourt, & Sorá, 2018). From this perspective, placing Chinese universities into international rankings is a political act rather than simply social science. Rankings, then, become an artifact of globalization reflecting political posturing.

It is important to tread lightly here because a vast literature is available addressing globalization. Certainly, Chinese universities are actively widening the scope of cross-border communication, intensifying transnational mobility, and, apparently, growing dependency on global structures—all features of globalization. Further, geopolitical power struggles may be served in different ways by rankings and their underlying assumptions, such as ones that present China’s universities as challenging those of the West for preeminence in knowledge creation for competitiveness or military hegemony. The rankings can also be considered as instantiating an unequal distribution of power that are at the root of asymmetrical power relations. These measures of symmetry or asymmetry, elite status, or reputation, then become important to challenge from several angles, of which we addressed one here.