<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Purpose</h3><p>Building on <xref ref-type="bibr" rid="j_jdis-2021-0014_ref_017">Leydesdorff, Bornmann, and Mingers (2019)</xref>, we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We address the question of whether differences are statistically significant in the rankings of Chinese universities. We propose methods for measuring statistical significance among different universities within or among countries.</p></sec>
<sec><h3>Design/methodology/approach</h3><p>Based on <italic>z</italic>-testing and overlapping confidence intervals, and using data about 205 Chinese universities included in the Leiden Rankings 2020, we argue that three main groups of Chinese research universities can be distinguished (low, middle, and high).</p></sec>
<sec><h3>Findings</h3><p>When the sample of 205 Chinese universities is merged with the 197 US universities included in Leiden Rankings 2020, the results similarly indicate three main groups: low, middle, and high. Using this data (Leiden Rankings and Web of Science), the <italic>z</italic>-scores of the Chinese universities are significantly below those of the US universities albeit with some overlap.</p></sec>
<sec><h3>Research limitations</h3><p>We show empirically that differences in ranking may be due to changes in the data, the models, or the modeling effects on the data. The scientometric groupings are not always stable when we use different methods.</p></sec>
<sec><h3>Practical implications</h3><p>Differences among universities can be tested for their statistical significance. The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings.</p></sec>
<sec><h3>Originality/value</h3><p>In the discussion about the rankings of universities, the question of whether differences are statistically significant, has, in our opinion, insufficiently been addressed in research evaluations.</p></sec>
</abstract>

PurposeBuilding on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We address the question of whether differences are statistically significant in the rankings of Chinese universities. We propose methods for measuring statistical significance among different universities within or among countries.
Design/methodology/approachBased on z-testing and overlapping confidence intervals, and using data about 205 Chinese universities included in the Leiden Rankings 2020, we argue that three main groups of Chinese research universities can be distinguished (low, middle, and high).
FindingsWhen the sample of 205 Chinese universities is merged with the 197 US universities included in Leiden Rankings 2020, the results similarly indicate three main groups: low, middle, and high. Using this data (Leiden Rankings and Web of Science), the z-scores of the Chinese universities are significantly below those of the US universities albeit with some overlap.
Research limitationsWe show empirically that differences in ranking may be due to changes in the data, the models, or the modeling effects on the data. The scientometric groupings are not always stable when we use different methods.
Practical implicationsDifferences among universities can be tested for their statistical significance. The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings.
Originality/valueIn the discussion about the rankings of universities, the question of whether differences are statistically significant, has, in our opinion, insufficiently been addressed in research evaluations.

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Amsterdam School of Communication Research (ASCoR)

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

PurposeBuilding on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We...

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Article Category: Research Paper

Published Online: Mar 02, 2021

Page range: 67 - 95

Received: Nov 01, 2020

Accepted: Jan 25, 2021

DOI: https://doi.org/10.2478/jdis-2021-0014

Keywords
Ranking, University, China, Significance of differences, Indicators

© 2021 Loet Leydesdorff et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Article Category: Research Paper

Published Online: Mar 02, 2021

Page range: 67 - 95

Received: Nov 01, 2020

Accepted: Jan 25, 2021

DOI: https://doi.org/10.2478/jdis-2021-0014

KeywordsRanking, University, China, Significance of differences, Indicators

© 2021 Loet Leydesdorff et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
Ranking, University, China, Significance of differences, Indicators