Research evaluation reform and the heterogeneity of researchers’ metric-wiseness

Figure 1.

Familiarity with calls to reform existing research evaluation practices_

Do you know…? (N = 263)	Yes, I know this		Yes, I have heard about this but do not know its content		No, I don’t know this
The DORA declaration	27	(10.3%)	29	(11.0%)	207	(78.7%)
The Leiden Manifesto	25	(9.5%)	41	(15.6%)	197	(74.9%)
Responsible metrics	14	(5.3%)	42	(16.0%)	207	(78.7%)
Metric Tide report	7	(2.7%)	22	(8.4%)	234	(89.0%)

Distribution of participants by function and gender compared with the distribution of the population of Sapienza University on December 31, 2018 (“Total N” and “%”)_ (Adapted from Rousseau et al_, 2021)_

Function	N. of resp.	% in sample	% sample compared to total N in category	Total N	%
Full professor	60	19.17	8.90	674	20.39
Associate professor	121	38.66	10.42	1,161	35.12
Full-time assistant professor	69	22.04	6.26	1,102	33.33
Contract professor (professori incaricati)	8	2.56	-	-	-
Temporary assistant professor L. 230/05	1	0.32	-	-	-
Temporary assistant professor L 240/10 Tipo A	23	7.35	10.85	212	6.41
Temporary assistant professor 240/10 Tipo B	16	5.11	10.19	157	4.75
Other (retired)	15	4.79	-	-	-
Gender
Male	199	63.58	10.04	1,982	59.95
Female	113	36.10	8.53	1,324	40.05
X	1	0.32		-	-

Cronbach’s alpha for the components capturing metric-wiseness_

		Cronbach’s alpha
Component 1	Technical knowledge of indicators	Not applicable (true - false statements)
Component 2	Use of indicators	0.3278
Component 3	Researchers’ intrinsic motivation	0.3735
Component 4	External pressure	0.6135

Component 3 statements were ordered in increasing levels of absolutely agreeing participants_

N = 263	Absolutely agree	Agree	Neutral	Disagree	Absolutely disagree
C3.S1 - If I do not have the expertise to solve a particular problem, I do not hesitate to ask a colleague to collaborate with me	14 (5.3%)	25 (9.5%)	68 (25.9%)	80 (30.4%)	76 (28.9%)
C3.S2 - I select research problems inspired by my own curiosity	29 (11.0%)	49 (18.6%)	48 (18.3%)	51 (19.4%)	86 (32.7%)
C3.S3 - I select topics for research based on their potential to advance science	82 (31.2%)	120 (45.6%)	49 (18.6%)	7 (2.7%)	5 (1.9%)

Component 2 was ordered by the increasing number of participants who agreed_

N = 263	Absolutely agree	Agree	Neutral	Disagree	Absolutely disagree
C2.S1 - Bibliometric indicators are equally useful in evaluating disciplinary and interdisciplinary research	7 (2.7%)	46 (17.5%	79 (30.0%)	87 (33.1%)	44 (16.7%)
C2.S2 - Besides citation-based indicators, one must, in applied fields, also take patent-based and similar indicators into account when evaluating researchers	22 (8.4%)	94 (35.7%)	108 (41.1%)	22 (8.4%)	17 (6.5%)
C2.S3 - The social influence of research must be taken into account in evaluating researchers	34 (12.9%)	81 (30.8%)	83 (31.6%)	45 (17.1%)	20 (7.6%)
C2.S4 - Besides citation-based indicators, one must also take journal standing within a field into account	41 (15.6%)	130 (49.4%)	64 (24.3%)	18 (6.8%)	10 (3.8%)
C2.S5 - A purely bureaucratic, automatic and quantitative approach to research evaluation is unbiased for an individual researcher	62 (23.6%)	39 (14.8%)	42 (16.0%)	47 (17.9%)	73 (27.8%)
C2.S6 - The quality of a researcher should be measured in relative terms within a field rather than in absolute terms	126 (47.9%)	108 (41.1%)	22 (8.4%)	4 (1.5%)	3 (1.1%)

Component 4 statements were ordered in increasing levels of absolutely agreeing participants_

N = 263	Absolutely agree	Agree	Neutral	Disagree	Absolutely disagree
C4.S1 - My institution influences how I communicate the results	10	52	87	70	44
of my research	(3.8%)	(19.8%)	(33.1%)	(26.6%)	(16.7%)
C4.S2 - I feel ‘publish or perish’ pressure in carrying out my	12	48	96	70	37
research	(4.6%)	(18.3%)	(36.5%)	(26.6%)	(14.1%)
C4.S3 - I select topics for research based on their potential to get	15	35	84	88	41
published quickly	(5.7%)	(13.3%)	(31.9%)	(33.5%)	(15.6%)
C4.S4 - It is important to use social media (Twitter, blogs…) to	27	104	88	36	8
share the results of my research	(10.3%)	(39.5%)	(33.5%)	(13.7%)	(3.0%)
C4.S5 - It is important to use academic research networks	30	90	65	59	19
(Mendeley, ResearchGate.) to share the results of my research	(11.4%)	(34.2%)	(24.7%)	(22.4%)	(7.2%)
C4.S6 - My likelihood of being promoted depends only on the	38	96	55	55	19
number of articles published in journals indexed in WoS or Scopus	(14.4%)	(36.5%)	(20.9%)	(20.9%)	(7.2%)
C4.S7 - ANVUR influences my publication strategies	41 (15.6%)	58 (22.1%)	51 (19.4%)	49 (18.6%)	64 (24.3%)
C4.S8 - Open Science (including publication, conservation and reuse of research data) is relevant for my research	42 (16.0%)	95 (36.1%)	101 (38.4%)	21 (8.0%)	4 (1.5%)
C4.S9 - My likelihood of being promoted depends mainly on the number of articles of which I am first or corresponding author	53 (20.2%)	104 (39.5%)	68 (25.9%)	32 (12.2%)	6 (2.3%)
C4.S10 - I feel completely free to publish my research in any way I want	99 (37.6%)	128 (48.7%)	27 (10.3%)	7 (2.7%)	2 (0.8%)
C4.S11 - The Ministry of Education and Research (MIUR) influences my publication strategies	107 (40.7%)	86 (32.7%)	32 (12.2%)	29 (11.0%)	9 (3.4%)
C4.S12 - My institution (Sapienza) influences my publication strategies	144 (54.8%)	92 (35.0%)	19 (7.2%)	6 (2.3%)	2 (0.8%)

The most important motivation for publishing (participants could select 3 out of 16 listed items)_

N = 313	N	%
To contribute to the scientific progress in your discipline	177	57%
To share your research findings with the academic community	146	47%
To improve your chances of receiving research funding	95	30%
Your personal intrinsic motivation	84	27%
To increase your chances to be promoted	69	22%
To improve your standing among your peers	55	18%
To help others (e.g. doctoral students, project collaborators…)	54	17%
To increase your probability of finding a new position	32	10%
To improve your standing in your current institution	30	10%
To make your current position permanent	26	8%
To increase the prestige and the resources allocated to your department	23	7%
To improve the standing of your institution	19	6%
To share your research findings with policymakers and practitioners	18	6%
To fulfill project requirements	16	5%
To fulfill administrative requirements	16	5%
To get a monetary reward	2	1%

Keyword search analysis of four documents of recent research evaluation reform initiatives_

Initiative document	Number of times the keyword “heterogeneity” or “diversity” or “variety” or “multiplicity” or “dissimilarity” is present in the document
DORA declaration (DORA, 2015)	Variety 1 time cited (“6. Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics”.)
Leiden Manifesto (Hicks et al., 2015)	No keywords cited
Hong Kong Declaration (Moher et al., 2020)	Variety 2 times cited but with a generic meaning of many not related to evaluation principles (“Selective publishing of research with positive results (i.e. publication bias) distorts science’s evidence base and has been demonstrated in a variety of disciplines including economics, psychology, and clinical and preclinical health research”; “The Center for Open Science’s Transparency and Openness Promotion initiative provides information on data transparency standards for a wide variety of discipline journals”.)Diversity 3 times cited (“We present five principles: responsible research practices; transparent reporting; open science (open research); valuing a diversity of types of research; and recognizing all contributions to research and scholarly activity”; “Some funders have already recognized the relevance of a broad range of research activities. The Research Impact Assessment Platform (Researchfish) works to capture some of this diversity and can generate reports on the impact of a broad spectrum of funded research”; “The HKPs do not address gender and other forms of diversity, inclusiveness, and related issues”.)
European Commission (2021) Scoping document	Variety 2 times cited (“Career assessment should take into account the variety of activities of academics such as teaching, research, entrepreneurship, management or leadership”; “To achieve excellent and relevant higher education, support is also needed to stimulate pedagogical innovation, focused on the learners, with a variety of learning spaces and flexible, interdisciplinary paths”.)Diversity 10 times cited (“Foster diversity, inclusiveness and gender equality”, “Develop a European framework for diversity and inclusion, including on gender gaps, identifying challenges and solutions for universities, and the needed support of public authorities”, “To encourage universities to implement institutional change through concrete measures for diversity and inclusion, including voluntary, quantified targets for inclusion and inclusive gender equality plans…”; “Universities are key to promote active citizenship, tolerance, equality and diversity, openness and critical thinking for more social cohesion and social trust, and thus protect European democracies”; “To support the diversity within the European higher education sector”; “The diversity and international standing of the EU education systems” the different types of higher education institutions are all hallmarks of our European way of life. This diversity is a strength, as it allows for choice and for creativity and synergy through mobility and cooperation”, “The European Union and Member States have a shared interest in supporting the higher education sector by joining their forces around a joint vision for the higher education sector, building on the richness of its diversity”; “Diversity, inclusiveness and gender equality in the higher education sector have become more important than ever”; “support universities as lighthouses of our European way of life:…2) diversity and inclusion”.)

Technical knowledge of indicators (ranked according to the number of correct answers)_

N = 263	True, I am sure	True, I think	I do not know	False, I think	False, I am sure	Correct answers
C1.S1 - Bibliometric indicators can easily be compared across disciplines [FALSE]	8 (3.0%)	21 (8.0%)	34 (12.9%)	95 (36.1%)	105 (39.9%)	200 (76.0%)
C1.S2 - Open Access journals never have a Web of Science impact factor [FALSE]	7 (2.7%)	16 (6.1%)	78 (29.7%)	82 (31.2%)	80 (30.4%)	162 (61.6%)
C1.S3 - On average older researchers have higher h-indices [TRUE]	54 (20.5%)	96 (36.5%)	52 (19.8%)	49 (18.6%)	12 (4.6%)	150 (57.0%)
C1.S4 - Citations received in conference proceedings are always included in an article’s total number of received citations in the Web of Science [FALSE]	12 (4.6%)	35 (13.3%)	117 (44.5%)	68 (25.9%)	31 (11.8%)	99 (37.6%)

Description of the dataset (Adapted from Rousseau et al_, 2021)_

Faculty	Group	Sample			Total Sapienza
		N	% in sample	% in category	N	%
Mathematics, Physics and Natural Sciences	Exact sciences	59	18.85	14.15	417	12.61
Architecture		9	2.88	5.36	168	5.08
Civil and Industrial Engineering		37	11.82	12.80	289	8.74
Information Engineering, Informatics and Statistics	Engineering & Technology	38	12.14	17.12	222	6.72
School of Aerospace Engineering		2	0.64	20.00	10	0.30
Pharmacy and Medicine		36	11.50	7.81	461	13.94
Medicine and Dentistry	Medical sciences	32	10.22	5.50	584	17.66
Medicine and Psychology		29	9.27	8.76	331	10.01
Arts and Humanities	Humanities and Social sciences	38	12.14	10.11	376	11.37
Economics		11	3.51	6.15	179	5.41
Law		4	1.28	4.65	86	2.60
Political Science, Sociology and Communication Science		18	5.75	9.84	183	5.54
TOTAL		313		9.47	3,306