<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec>
<h3>Purpose</h3>
<p>This study examines the effects of using publication-based metrics for the initial screening in the application process for a project leader. The key questions are whether formal policy affects the allocation of funds to researchers with a better publication record and how the previous academic performance of principal investigators is related to future project results.</p>
</sec>
<sec>
<h3>Design/methodology/approach</h3>
<p>We compared two competitions, before and after the policy raised the publication threshold for the principal investigators. We analyzed 9,167 papers published by 332 winners in physics and the social sciences and humanities (SSH), and 11,253 publications resulting from each funded project.</p>
</sec>
<sec>
<h3>Findings</h3>
<p>We found that among physicists, even in the first period, grants tended to be allocated to prolific authors publishing in high-quality journals. In contrast, the SSH project grantees had been less prolific in publishing internationally in both periods; however, in the second period, the selection of grant recipients yielded better results regarding awarding grants to more productive authors in terms of the quantity and quality of publications. There was no evidence that this better selection of grant recipients resulted in better publication records during grant realization.</p>
</sec>
<sec>
<h3>Originality</h3>
<p>This study contributes to the discussion of formal policies that rely on metrics for the evaluation of grant proposals. The Russian case shows that such policy may have a profound effect on changing the supply side of applicants, especially in disciplines that are less suitable for metric-based evaluations. In spite of the criticism given to metrics, they might be a useful additional instrument in academic systems where professional expertise is corrupted and prevents allocation of funds to prolific researchers.</p>
</sec>
</abstract>

Purpose
This study examines the effects of using publication-based metrics for the initial screening in the application process for a project leader. The key questions are whether formal policy affects the allocation of funds to researchers with a better publication record and how the previous academic performance of principal investigators is related to future project results.


Design/methodology/approach
We compared two competitions, before and after the policy raised the publication threshold for the principal investigators. We analyzed 9,167 papers published by 332 winners in physics and the social sciences and humanities (SSH), and 11,253 publications resulting from each funded project.


Findings
We found that among physicists, even in the first period, grants tended to be allocated to prolific authors publishing in high-quality journals. In contrast, the SSH project grantees had been less prolific in publishing internationally in both periods; however, in the second period, the selection of grant recipients yielded better results regarding awarding grants to more productive authors in terms of the quantity and quality of publications. There was no evidence that this better selection of grant recipients resulted in better publication records during grant realization.


Originality
This study contributes to the discussion of formal policies that rely on metrics for the evaluation of grant proposals. The Russian case shows that such policy may have a profound effect on changing the supply side of applicants, especially in disciplines that are less suitable for metric-based evaluations. In spite of the criticism given to metrics, they might be a useful additional instrument in academic systems where professional expertise is corrupted and prevents allocation of funds to prolific researchers.

Evaluating grant proposals: lessons from using metrics as screening device

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution 4.0 International License.

{"article-title":"Evaluating grant proposals: lessons from using metrics as screening device"}

Purpose
This study examines the effects of using publication-based metrics for the initial screening in the application process for a project leader. The key...

Evaluating grant proposals: lessons from using metrics as screening device

Article Category: Research Paper

Pubblicato online: 07 giu 2023

Pagine: 66 - 92

Ricevuto: 19 dic 2022

Accettato: 13 apr 2023

DOI: https://doi.org/10.2478/jdis-2023-0010

© 2023 Katerina Guba et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.