Citation distributions and research evaluations: The impossibility of formulating a universal indicator

Abramo, G., Cicero, T., & D’Angelo, C. A. (2013). National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case. Scientometrics, 95, 311-324. Abramo G. Cicero T. D’Angelo C. A. , ( 2013 ). National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case . Scientometrics , 95 , 311 – 324 . Search in Google Scholar

Abramo, G., & D’Angelo, C. A. (2016a). A farewell to the MNCS and like size-independent indicators: Rejoinder. Journal of Informetrics, 10(2), 679-683. Abramo G. D’Angelo C. A. , ( 2016a ). A farewell to the MNCS and like size-independent indicators: Rejoinder . Journal of Informetrics , 10 ( 2 ), 679 – 683 . Search in Google Scholar

Abramo, G., & D’Angelo, C. A. (2016b). A farewell to the MNCS and like size-indpendent indicators. Journal of Informetrics, 10(2), 646-651. Abramo G. D’Angelo C. A. , ( 2016b ). A farewell to the MNCS and like size-indpendent indicators . Journal of Informetrics , 10 ( 2 ), 646 – 651 . Search in Google Scholar

Abramo, G., D’Angelo, C. A., & Reale, E. (2019). Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications? Scientometrics, 121, 537-554. Abramo G. D’Angelo C. A. Reale E. , ( 2019 ). Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications? Scientometrics , 121 , 537 – 554 . Search in Google Scholar

Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1), 2158244019829575. Aksnes D. W. Langfeldt L. Wouters P. , ( 2019 ). Citations, citation indicators, and research quality: An overview of basic concepts and theories . SAGE Open , 9 ( 1 ), 2158244019829575 . Search in Google Scholar

Aksnes, D. W., Piro, F. N., & Fossum, L. W. (2023). Citation metrics covary with researchers’ assessments of the quality of their works. Quantitative Science Studies, 4(1), 105-126. Aksnes D. W. Piro F. N. Fossum L. W. , ( 2023 ). Citation metrics covary with researchers’ assessments of the quality of their works . Quantitative Science Studies , 4 ( 1 ), 105 – 126 . Search in Google Scholar

Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2010). A comparison of the scientific performance of the U.S. and the European Union at the turn of the 21st century. Scientometrics, 85(1), 329-344. Albarrán P. Crespo J. A. Ortuño I. Ruiz-Castillo J. , ( 2010 ). A comparison of the scientific performance of the U.S. and the European Union at the turn of the 21st century . Scientometrics , 85 ( 1 ), 329 – 344 . Search in Google Scholar

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information science and technology, 64(2), 217-233 Bornmann L. , ( 2013 ). What is societal impact of research and how can it be assessed? A literature survey . Journal of the American Society for information science and technology , 64 ( 2 ), 217233 . Search in Google Scholar

Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158-165. Bornmann L. Leydesdorff L. Mutz R. , ( 2013 ). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits . Journal of Informetrics , 7 ( 1 ), 158 – 165 . Search in Google Scholar

Bornmann, L., Ye, A., & Ye, F. (2018). Identifying landmark publications in the long run using field-normalized citation data. Journal of Documentation, 74(2), 278-288. Bornmann L. Ye A. Ye F. , ( 2018 ). Identifying landmark publications in the long run using field-normalized citation data . Journal of Documentation , 74 ( 2 ), 278 – 288 . Search in Google Scholar

Boylan, G. L., & Cho, B. R. (2012). The normal probability plot as a tool for understanding data: A shape analysis from the perspective of skewness, kurtosis, and variability. Quality and Reliability Engineering International, 28(3), 249-264. Boylan G. L. Cho B. R. , ( 2012 ). The normal probability plot as a tool for understanding data: A shape analysis from the perspective of skewness, kurtosis, and variability . Quality and Reliability Engineering International , 28 ( 3 ), 249 – 264 . Search in Google Scholar

Bozeman, B., & Sarewitz, D. (2011). Public value mapping and science policy evaluation. Minerva, 49, 1-23. Bozeman B. Sarewitz D. , ( 2011 ). Public value mapping and science policy evaluation . Minerva , 49 , 1 – 23 . Search in Google Scholar

Dosi, G., Llerena, P., & Labini, M. S. (2006). The relationships between science, technologies and their industrial exploitation: An illustration through the myths and realities of the so-called ‘European Paradox’. Research Policy, 35(10), 1450-1464. Dosi G. Llerena P. Labini M. S. , ( 2006 ). The relationships between science, technologies and their industrial exploitation: An illustration through the myths and realities of the so-called ‘European Paradox’ . Research Policy , 35 ( 10 ), 1450 – 1464 . Search in Google Scholar

European Commission. (2022). Science, Reserach and Innovation Performance of the EU. Building a sustainable future in uncertain times. European Commision. European Commission . ( 2022 ). Science, Reserach and Innovation Performance of the EU. Building a sustainable future in uncertain times . European Commision . Search in Google Scholar

Garfield, E., & Welljams-Dorof, A. (1992). Citation data: Their use as quantitative indicators for science and technology evaluations and policy-making. Science and Public Policy, 19(5), 321-327. Garfield E. Welljams-Dorof A. , ( 1992 ). Citation data: Their use as quantitative indicators for science and technology evaluations and policy-making . Science and Public Policy , 19 ( 5 ), 321 – 327 . Search in Google Scholar

Godin, B. (2006). On the origins of bibliometrics. Scientometrics, 68(1), 109-133. Godin B. , ( 2006 ). On the origins of bibliometrics . Scientometrics , 68 ( 1 ), 109 – 133 . Search in Google Scholar

Greenhalgh, T., Raftery, J., Hanney, S., & Glover, M. (2016). Research impact: A narrative review. BMC Medicine, 14, 1-16. Greenhalgh T. Raftery J. Hanney S. Glover M. , ( 2016 ). Research impact: A narrative review . BMC Medicine , 14 , 1 – 16 . Search in Google Scholar

Harnad, S. (2009). Open access scientometrics and the UK research assessment exercise. Scientometrics, 79, 147-156. Harnad S. , ( 2009 ). Open access scientometrics and the UK research assessment exercise . Scientometrics , 79 , 147 – 156 . Search in Google Scholar

Kuhn, T. (1970). The structure of scientific revolutions. Chicago: University of Chicago Press. Kuhn T. , ( 1970 ). The structure of scientific revolutions . Chicago : University of Chicago Press . Search in Google Scholar

Leydesdorff, L. (2005). Evaluation of research and evolution of science indicators. Current Science, 89(9), 1510-1517. Leydesdorff L. , ( 2005 ). Evaluation of research and evolution of science indicators . Current Science , 89 ( 9 ), 1510 – 1517 . Search in Google Scholar

Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36, 343-362. Martin B. R. , ( 1996 ). The use of multiple indicators in the assessment of basic research . Scientometrics , 36 , 343 – 362 . Search in Google Scholar

Martin, B. R. (2011). The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247-254. Martin B. R. , ( 2011 ). The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation , 20 ( 3 ), 247 – 254 . Search in Google Scholar

Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill, NJ: Computer Horizon Inc. Narin F. , ( 1976 ). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity . Cherry Hill, NJ : Computer Horizon Inc . Search in Google Scholar

National Institute of Science and Technology Policy (2022). Japanese science and technology indicators 2022(NISTEP research material No. 318). Tokyo: National Institute of Science and Technology Policy. https://doi.org/10.15108/m318e National Institute of Science and Technology Policy , ( 2022 ). Japanese science and technology indicators 2022(NISTEP research material No. 318) . Tokyo : National Institute of Science and Technology Policy . https://doi.org/10.15108/m318e Search in Google Scholar

Normile, D. (2024). Japan tries, again, to boost global ranking of its universities. Science, 383(6678), 12-13 Normile D. , ( 2024 ). Japan tries, again, to boost global ranking of its universities . Science , 383 ( 6678 ), 12 – 13 Search in Google Scholar

OECD. (2016). OECD Science, Technology and Innovation Outlook 2016. Paris: OECD Publishing. https://doi.org/10.1787/sti_in_outlook-2016-en. OECD . ( 2016 ). OECD Science, Technology and Innovation Outlook 2016 . Paris : OECD Publishing . https://doi.org/10.1787/sti_in_outlook-2016-en. Search in Google Scholar

Pendlebury, D. A. (2020). When the data don’t mean what they say: Japan’s comparative underperformance in citation impact. In C. Daraio & W. Glanzel (Eds.), Evaluative Informetrics: The Art of Metrics-based Research Assessment. Spriger. Pendlebury D. A. , ( 2020 ). When the data don’t mean what they say: Japan’s comparative underperformance in citation impact . In Daraio C. Glanzel W. (Eds.), Evaluative Informetrics: The Art of Metrics-based Research Assessment . Spriger . Search in Google Scholar

Perianes-Rodriguez, A., & Ruiz-Castillo, J. (2016). University citation distribution. Journal of the Association for Information Science and Technology, 67(11), 2790-2804. Perianes-Rodriguez A. Ruiz-Castillo J. , ( 2016 ). University citation distribution . Journal of the Association for Information Science and Technology , 67 ( 11 ), 2790 – 2804 . Search in Google Scholar

Poege, F., Harhoff, D., Gaessler, F., & Baruffaldi, S. (2019). Science quality and the value of inventions. Science Advances, 5(12), eaay7323. Poege F. Harhoff D. Gaessler F. Baruffaldi S. , ( 2019 ). Science quality and the value of inventions . Science Advances , 5 ( 12 ), eaay7323 . Search in Google Scholar

Rodríguez-Navarro, A. (2024a). Countries pushing the boundaries of knowledge: The USA’s dominance, China’s rise, and the EU’s stagnation. Preprint at arXiv:2402.15263. Rodríguez-Navarro A. , ( 2024a ). Countries pushing the boundaries of knowledge: The USA’s dominance, China’s rise, and the EU’s stagnation . Preprint at arXiv:2402.15263 . Search in Google Scholar

Rodríguez-Navarro, A. (2024b). Uncertain research country rankings. Should we continue producing uncertain rankings? Preprint at arXiv:2312.17560v2. Rodríguez-Navarro A. , ( 2024b ). Uncertain research country rankings. Should we continue producing uncertain rankings? Preprint at arXiv:2312.17560v2 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2018). Double rank analysis for research assessment. Journal of Informetrics, 12(1), 31-41. Rodríguez-Navarro A. Brito R. , ( 2018 ). Double rank analysis for research assessment . Journal of Informetrics , 12 ( 1 ), 31 – 41 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2019). Probability and expected frequency of breakthroughs: Basis and use of a robust method of research assessment. Scientometrics, 119(1), 213-235. Rodríguez-Navarro A. Brito R. , ( 2019 ). Probability and expected frequency of breakthroughs: Basis and use of a robust method of research assessment . Scientometrics , 119 ( 1 ), 213 – 235 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2020a). Like-for-like bibliometric substitutes for peer review: Advantages and limits of indicators calculated from the ep index. Research Evaluation, 29(2), 215-230. Rodríguez-Navarro A. Brito R. , ( 2020a ). Like-for-like bibliometric substitutes for peer review: Advantages and limits of indicators calculated from the ep index . Research Evaluation , 29 ( 2 ), 215 – 230 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2020b). Might Europe one day again be a global scientific powerhouse? Analysis of ERC publications suggests it will not be possible without changes in research policy. Quantitative Science Studies, 1(2), 872-893. Rodríguez-Navarro A. Brito R. , ( 2020b ). Might Europe one day again be a global scientific powerhouse? Analysis of ERC publications suggests it will not be possible without changes in research policy . Quantitative Science Studies , 1 ( 2 ), 872 – 893 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2021). Total number of papers and in a single percentile fully describes research impact-Revisiting concepts and applications. Quantitative Science Studies, 2(2), 544-559. Rodríguez-Navarro A. Brito R. , ( 2021 ). Total number of papers and in a single percentile fully describes research impact-Revisiting concepts and applications . Quantitative Science Studies , 2 ( 2 ), 544 – 559 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2022). The link between countries’ economic and scientific wealth has a complex dependence on technological activity and research policy. Scientometrics, 127(5), 2871-2896. Rodríguez-Navarro A. Brito R. , ( 2022 ). The link between countries’ economic and scientific wealth has a complex dependence on technological activity and research policy . Scientometrics , 127 ( 5 ), 2871 – 2896 . Search in Google Scholar

Rodríguez-Navarro, A., & Brito, R. (2024). Rank analysis of most cited publications, a new approach for research assessments. Journal of Informetrics, 18(2), 101503. Rodríguez-Navarro A. Brito R. , ( 2024 ). Rank analysis of most cited publications, a new approach for research assessments . Journal of Informetrics , 18 ( 2 ), 101503 . Search in Google Scholar

Science and Technology Observatory (OST) (2019). Dynamics of scientific production in the world, in Europe and France, 2000-2016. Paris: Hcéres. Science and Technology Observatory (OST) , ( 2019 ). Dynamics of scientific production in the world, in Europe and France, 2000-2016 . Paris : Hcéres . Search in Google Scholar

Sivertsen, G. (2018). The Norwegian model in Norway. Journal of Data and Information Science, 3(4), 3-19. Sivertsen G. , ( 2018 ). The Norwegian model in Norway . Journal of Data and Information Science , 3 ( 4 ), 3 – 19 . Search in Google Scholar

Tahamtan, I., & Bornmann, L. (2019). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics, 121(3), 1635-1684. Tahamtan I. Bornmann L. , ( 2019 ). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018 . Scientometrics , 121 ( 3 ), 1635 – 1684 . Search in Google Scholar

Thelwall, M., Kousha, K., Stuart, E., Makita, M., Abdoli, M., PWilson, & Levitt, J. (2023). In which fields are citations indicators of reserach quality? Journal of the Association for Information Science and Technology, 74(8), 941-953. Thelwall M. Kousha K. Stuart E. Makita M. Abdoli M. PWilson Levitt J. , ( 2023 ). In which fields are citations indicators of reserach quality? Journal of the Association for Information Science and Technology , 74 ( 8 ), 941 – 953 . Search in Google Scholar

Traag, V. A., & Waltman, L. (2019). Systematic analysis of agreement between metrics and peer review in the UK REF. Palgrave Communications, 5(1), 29. https://doi.org/10.1057/s41599-019-0233-x Traag V. A. Waltman L. , ( 2019 ). Systematic analysis of agreement between metrics and peer review in the UK REF . Palgrave Communications , 5 ( 1 ), 29 . https://doi.org/10.1057/s41599-019-0233-x Search in Google Scholar

Van Noorden, R. (2010). Metrics: A profusion of measures. Nature, 465(7300), 864-866. Van Noorden R. , ( 2010 ). Metrics: A profusion of measures . Nature , 465 ( 7300 ), 864 – 866 . Search in Google Scholar

Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365-391. Waltman L. , ( 2016 ). A review of the literature on citation impact indicators . Journal of Informetrics , 10 ( 2 ), 365 – 391 . Search in Google Scholar

Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van-Eck, N. J., van-Leeuwen, T. N., van-Raan, A. F. J., Visser, M. S., & Wouters, P. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for information Science and Technology, 63(12), 2419-2432. Waltman L. Calero-Medina C. Kosten J. Noyons E. C. M. Tijssen R. J. W. van-Eck N. J. van-Leeuwen T. N. van-Raan A. F. J. Visser M. S. Wouters P. , ( 2012 ). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation . Journal of the American Society for information Science and Technology , 63 ( 12 ), 2419 – 2432 . Search in Google Scholar

Waltman, L., & Schreiber, M. (2013). On the calculation of percentile-based bibliometric indicators. Journal of the American Society for information Science and Technology, 64(2), 372-379. Waltman L. Schreiber M. , ( 2013 ). On the calculation of percentile-based bibliometric indicators . Journal of the American Society for information Science and Technology , 64 ( 2 ), 372 – 379 . Search in Google Scholar

Wildgaard, L., Schneider, J. W., & Larsen, B. (2014). A review of the characteristics of 108 authorlevel bibliometric indicators. Scientometrics, 101, 125-158. Wildgaard L. Schneider J. W. Larsen B. , ( 2014 ). A review of the characteristics of 108 authorlevel bibliometric indicators . Scientometrics , 101 , 125 – 158 . Search in Google Scholar