Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Alsmadi, I., & Taylor, Z. (2018). Examining university ranking metrics: Articulating issues of size and web dependency. In Proceedings of the 2018 International Conference on Computing and Big Data (ICCBD ’18). Association for Computing Machinery, New York, NY, USA, 73–77. doi: https://doi.org/10.1145/3277104.3277111 AlsmadiI. TaylorZ. 2018 Examining university ranking metrics: Articulating issues of size and web dependency In Proceedings of the 2018 International Conference on Computing and Big Data (ICCBD ’18) Association for Computing Machinery, New York, NY, USA 73 77 https://doi.org/10.1145/3277104.3277111 Open DOISearch in Google Scholar

Anderson, J., Collins, P.M.D., Irvine, J., Isard, P.A., Martin, B.R., Narin, F., & Stevens, K. (1988). On-line approaches to measuring national scientific output: A cautionary tale. Science and Public Policy 15(3), 153–161. AndersonJ. CollinsP.M.D. IrvineJ. IsardP.A. MartinB.R. NarinF. StevensK. 1988 On-line approaches to measuring national scientific output: A cautionary tale Science and Public Policy 15 3 153 161 Search in Google Scholar

Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286–291. BornmannL. LeydesdorffL. 2013 The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 Journal of Informetrics 7 2 286 291 10.1016/j.joi.2012.12.003 Search in Google Scholar

Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1), 228–230. BornmannL. MutzR. 2011 Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization Journal of Informetrics 5 1 228 230 10.1016/j.joi.2010.10.009 Search in Google Scholar

Bourdieu, P. (1998). The state nobility: Elite schools in the field of power. Stanford University Press. BourdieuP. 1998 The state nobility: Elite schools in the field of power Stanford University Press Search in Google Scholar

Bowman, N.A., & Bastedo, M.N. (2011). Anchoring effects in world university rankings: Exploring biases in reputation scores. Higher Education, 61(4), 431–444. BowmanN.A. BastedoM.N. 2011 Anchoring effects in world university rankings: Exploring biases in reputation scores Higher Education 61 4 431 444 10.1007/s10734-010-9339-1 Search in Google Scholar

Brewer, D.J., Gates, S.M., & Goldman, C.A. (2001). In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education. Piscataway, NJ: Transaction Publishers, Rutgers University. BrewerD.J. GatesS.M. GoldmanC.A. 2001 In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education Piscataway, NJ Transaction Publishers, Rutgers University Search in Google Scholar

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. CohenJ. 1988 Statistical Power Analysis for the Behavioral Sciences 2nd ed. Hillsdale, NJ Lawrence Erlbaum Search in Google Scholar

Colliander, C., & Ahlgren, P. (2011). The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments. Journal of Informetrics, 5(1), 101–113. CollianderC. AhlgrenP. 2011 The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments Journal of Informetrics 5 1 101 113 10.1016/j.joi.2010.09.003 Search in Google Scholar

Cumming, G. (2013). Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY/Hove, East Sussex, UK: Routledge. CummingG. 2013 Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis New York, NY/Hove, East Sussex, UK Routledge 10.4324/9780203807002 Search in Google Scholar

DiMaggio, P.J., & Powell, W.W. (1983). The Iron Cage Revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147–160. DiMaggioP.J. PowellW.W. 1983 The Iron Cage Revisited: Institutional isomorphism and collective rationality in organizational fields American Sociological Review 48 147 160 10.2307/j.ctv1f886rp.7 Search in Google Scholar

Garfield, E. (1971). The mystery of the transposed journal lists—wherein Bradford’s Law of Scattering is generalized according to Garfield’s Law of Concentration. Current Contents, 3(33), 5–6. GarfieldE. 1971 The mystery of the transposed journal lists—wherein Bradford’s Law of Scattering is generalized according to Garfield’s Law of Concentration Current Contents 3 33 5 6 Search in Google Scholar

Gingras, Y. (2016). Bibliometrics and research evaluation: An overview. Cambridge MA: MIT Press. GingrasY. 2016 Bibliometrics and research evaluation: An overview Cambridge MA MIT Press 10.7551/mitpress/10719.001.0001 Search in Google Scholar

Halffman, W., & Leydesdorff, L. (2010). Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities. Minerva, 48(1), 55–72. HalffmanW. LeydesdorffL. 2010 Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities Minerva 48 1 55 72 10.1007/s11024-010-9141-3285052520401157 Search in Google Scholar

Heilbron, J., Boncourt, T., & Sorá, G. (2018). Introduction: The Social and Human Sciences in Global Power Relations. In The Social and Human Sciences in Global Power Relations (pp. 1–25). Pal. grave Macmillan, Cham. HeilbronJ. BoncourtT. SoráG. 2018 Introduction: The Social and Human Sciences in Global Power Relations In The Social and Human Sciences in Global Power Relations 1 25 Pal. grave Macmillan, Cham. 10.1007/978-3-319-73299-2_1 Search in Google Scholar

Leydesdorff, L., & Wagner, C. (2009). Macro-level indicators of the relations between research funding and research output. Journal of Informetrics, 3(4), 353–362. doi: 10.1016/j.joi.2009.05.005 LeydesdorffL. WagnerC. 2009 Macro-level indicators of the relations between research funding and research output Journal of Informetrics 3 4 353 362 10.1016/j.joi.2009.05.005 Open DOISearch in Google Scholar

Leydesdorff, L., Bornmann, L., & Mingers, J. (2019). Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the leiden rankings. Journal of the Association for Information Science and Technology, 70(5), 509–525. LeydesdorffL. BornmannL. MingersJ. 2019 Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the leiden rankings Journal of the Association for Information Science and Technology 70 5 509 525 10.1002/asi.24130 Search in Google Scholar

Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the Tables on Citation Analysis One More Time: Principles for Comparing Sets of Documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381. LeydesdorffL. BornmannL. MutzR. OpthofT. 2011 Turning the Tables on Citation Analysis One More Time: Principles for Comparing Sets of Documents Journal of the American Society for Information Science and Technology 62 7 1370 1381 10.1002/asi.21534 Search in Google Scholar

Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators-a state-of-theart report. Scientometrics, 109(3), 2129–2150. LeydesdorffL. WoutersP. BornmannL. 2016 Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators-a state-of-theart report Scientometrics 109 3 2129 2150 10.1007/s11192-016-2150-8512404427942086 Search in Google Scholar

McAllister, P.R., Narin, F., & Corrigan, J.G. (1983). Programmatic Evaluation and Comparison Based on Standardized Citation Scores. IEEE Transactions on Engineering Management, 30(4), 205–211. McAllisterP.R. NarinF. CorriganJ.G. 1983 Programmatic Evaluation and Comparison Based on Standardized Citation Scores IEEE Transactions on Engineering Management 30 4 205 211 10.1109/TEM.1983.6448622 Search in Google Scholar

Peters, M.A. (2017). Global university rankings: Metrics, performance, governance. Educational Philosophy & Theory, 51(1), 1–9. PetersM.A. 2017 Global university rankings: Metrics, performance, governance Educational Philosophy & Theory 51 1 1 9 10.1080/00131857.2017.1381472 Search in Google Scholar

Schneider, J. (2015). Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102(1), 411–432. SchneiderJ. 2015 Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations Scientometrics 102 1 411 432 10.1007/s11192-014-1251-5 Search in Google Scholar

Sheskin, D.J. (2011). Handbook of Parametric and Nonparametric Statistical Procedures (5th Edition). Boca Raton, FL: Chapman & Hall/CRC. SheskinD.J. 2011 Handbook of Parametric and Nonparametric Statistical Procedures 5th Edition Boca Raton, FL Chapman & Hall/CRC Search in Google Scholar

Shin, J.C., Toutkoushian, R.K., & Teichler, U. (2011). University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education. Dordrecht: Springer. ShinJ.C. ToutkoushianR.K. TeichlerU. 2011 University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education Dordrecht Springer 10.1007/978-94-007-1116-7 Search in Google Scholar

Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics, 107(2), 357–368. SivertsenG. 2016 Patterns of internationalization and criteria for research assessment in the social sciences and humanities Scientometrics 107 2 357 368 10.1007/s11192-016-1845-1483382227122643 Search in Google Scholar

Sivertsen, G., Rousseau, R., & Zhang, L. (2019). Measuring scientific contributions with modified fractional counting. Journal of Informetrics, 13(2), 679–694. SivertsenG. RousseauR. ZhangL. 2019 Measuring scientific contributions with modified fractional counting Journal of Informetrics 13 2 679 694 10.1016/j.joi.2019.03.010 Search in Google Scholar

Tijssen, R.J.W., Visser, M.S., & van Leeuwen, T.N. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381–397. TijssenR.J.W. VisserM.S. van LeeuwenT.N. 2002 Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics 54 3 381 397 10.1023/A:1016082432660 Search in Google Scholar

van Raan, A.F.J., van Leeuwen, T.N., Visser, M.S., van Eck, N.J., & Waltman, L. (2010). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435. van RaanA.F.J. van LeeuwenT.N. VisserM.S. van EckN.J. WaltmanL. 2010 Rivals for the crown: Reply to Opthof and Leydesdorff Journal of Informetrics 4 3 431 435 10.1016/j.joi.2010.03.008 Search in Google Scholar

Waltman, L. (2016). Conceptual difficulties in the use of statistical inference in citation analysis. Journal of Informetrics, 4(10), 1249–1252. WaltmanL. 2016 Conceptual difficulties in the use of statistical inference in citation analysis Journal of Informetrics 4 10 1249 1252 10.1016/j.joi.2016.09.012 Search in Google Scholar

Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R.J., Eck, N.J., . . . Wouters, P. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432. WaltmanL. Calero-MedinaC. KostenJ. NoyonsE. TijssenR.J. EckN.J. WoutersP. 2012 The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation Journal of the American Society for Information Science and Technology 63 12 2419 2432 10.1002/asi.22708 Search in Google Scholar

Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., & Van Raan, A.F.J. (2011). Towards a New Crown Indicator: Some Theoretical Considerations. Journal of Informetrics, 5(1), 37–47. WaltmanL. Van EckN.J. Van LeeuwenT.N. VisserM.S. Van RaanA.F.J. 2011 Towards a New Crown Indicator: Some Theoretical Considerations Journal of Informetrics 5 1 37 47 10.1016/j.joi.2010.08.001 Search in Google Scholar

Wasserstein, R.L., & Lazar, N.A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. WassersteinR.L. LazarN.A. 2016 The ASA’s statement on p-values: Context, process, and purpose The American Statistician 70 2 129 133 10.1080/00031305.2016.1154108 Search in Google Scholar

Whitley, R.D. (1984). The Intellectual and Social Organization of the Sciences. Oxford: Oxford University Press. WhitleyR.D. 1984 The Intellectual and Social Organization of the Sciences Oxford Oxford University Press Search in Google Scholar