Evaluating grant proposals: lessons from using metrics as screening device

Researchers have recognized that grant funding plays a more significant role in supporting science compared to institutional funding (Auranen & Nieminen, 2010; Grimpe, 2012; Maisano et al., 2020; Wang et al., 2020). Grant funding is expected to trigger competition among scientists and thus lead to more significant academic results (Grimpe, 2012; Maisano et al., 2020). However, multiple studies have indicated that the effect of grant funding on research performance has been relatively small (Fedderke & Goldschmidt, 2015; Hornbostel et al., 2009; Jacob & Lefgren, 2011; Maisano et al., 2020; Morrillo, 2019; Paudel et al., 2020; Wang & Shapira, 2015). Investigators have partially attributed this to the peculiarities of the peer review process in proposal evaluation, which does not always lead to the selection of the best scientists (Fang et al., 2016; Graves et al., 2011; Gyorffy et al., 2018). The findings that the past scientometric performance of the proposal’s leading scientist as the best predictor of future project output has called for a formal policy of using metrics in deciding which projects to fund (Fedderke & Goldschmidt, 2015). However, the suggestion to rely on quantitative performance indicators in evaluating grant proposals has remained disputable, as the precedents are rare and we do not have sufficient information on how such policy may affect grant allocation. Can the process for evaluating applications be improved so that grants are awarded to more capable scientists? The present study contributes to the discussion by analyzing the effects of a formal policy related to a rising publication threshold for principal investigators between two waves of grant competitions: the first wave of grants was awarded in 2014 and the second wave in 2017. We present the analysis of empirical evidence from the Russian Science Foundation (RSF) that has developed publication metrics as an initial application threshold.

The question of whether using scientometric indicators changes academic evaluation results by allocating resources to top-performing researchers is especially relevant for developing countries, given deep distrust in academia and evidence of academic nepotism (Denisova-Schmidt, 2023; Sokolov, 2021). In this context, academic bureaucracy tends to consider scientometric indicators as ‘hard’ evidence that improves resource allocation by redistributing resources from corrupted academic elites to productive authors and organizations. However, on the academic periphery, the policy rationale of using metrics to provide better selection faces challenges related to weak ethical standards. There is a strong possibility that institutions and individuals might respond strategically by working on indicators at the expense of the quality of research results. Evidence shows that Russian universities have faced difficulties in increasing their international publications; these challenges have led the institutions to employ questionable strategies, including publishing in predatory journals (Guskov et al., 2018). Russia is included in the top-8 countries with the largest numbers of publications in potentially predatory journals (Marina & Sterligov, 2021). Data shows that the share of Russian output in Scopus in predatory journals was the highest in 2015 and 2016 – 8.23 and 8.4 while, in 2012, it was only 0.6 (Marina & Sterligov, 2021). Is it possible that in the allocation of funds metrics do not provide a barrier to researchers who behave opportunistically by choosing low-quality journals that are nevertheless indexed in international databases?

From the start of its operations, the Russian Science Foundation has relied on bibliometric indicators in proposal evaluations, using scientometric requirements as a barrier for applicants. In other words, to enter the grant competition, a scientist must meet the minimum publication threshold, which has remarkably increased over the past five years. The changes for scientists in the social sciences and humanities (SSH) were particularly dramatic, as the national scientific journals (those not indexed on international databases) disappeared from the list of approved sources in meeting the threshold. Starting from 2017, for researchers from all scientific fields, only publications that appeared in sources indexed by Scopus or Web of Science had been counted as acceptable output. This policy change allows us to study whether the reliance on metrics in the project evaluation procedure has facilitated a more effective selection of best-performing scientists. Given that both scholars with international publications and questionable publication records were eligible to apply for the grant competition, we can study whether the results of the selection process were in favor of those who were successful in internationalization or the emphasis on international databases in scientometric requirements have not changed the selection process.

Our study gives additional empirical evidence on the relationship between grant recipients’ past academic performance and future research project results. We explore the following considerations in greater detail: 1) whether the policy in question has affected the allocation of funds to researchers with a better publication record; 2) how principal investigators’ previous academic performance relates to future project results; 3) whether the link between previous and future performance increases as the role of publication-based metrics grows; and 4) whether differences in policy effects between disciplines can be observed. Our main empirical focus is on the change in the quality of the publication output of two grant-holder groups. The change in the raw number of publications would signal the change in the submission eligibility. However, the change in the quality of publication output would allow us to conclude whether the raising scientometric requirements resulted in the selection of the project leaders who were successful in publishing internationally or, rather, grants were distributed to authors with indexed publications but were less prolific in the production of global science. We acknowledge that scientometric indicators for the research evaluation can have different effects in the social and natural sciences. Therefore, we compare a group of project leaders in two areas of science separately: physics and the social sciences and humanities (SSH).

Our main dataset comprises 209 winners in 2014 who had published 5,701 papers altogether in the five years before submitting their application, and 123 winners in 2017 who had published 3,466 papers within the same time span. We also obtained information from the list of publications resulting from each funded project located on the foundation’s website (4604 publications for the first wave and 6649 publications for the second wave).

The body of literature on the efficiency of public funding is extensive and has concentrated particularly on evaluating the results of funded projects, measured by the quality and quantity of academic research output. Much of the research published has focused on the analysis of various foundations by tracking publications that resulted in supported projects (Gush et al., 2018; Gyorffy et al., 2020; Li & Agha, 2014). In studying foundations’ performance in this respect, a common approach has been to compare the performance of scholars who have received grants with that of similar authors without a grant (Fedderke & Goldschmidt, 2015).

Empirical studies on individual foundations have shown contradictory results. An analysis of publications by grant recipients in New Zealand revealed an increase in publications by between 3 and 15%, and citations by between 5 and 26% (Gush et al., 2018). Other researchers collected data on the performance measures of those who received additional financial resources and those who did not receive those (Fedderke & Goldschmidt, 2015). But while the comparison indicated that funding tended to lead to higher performance, the increase in publications could be modest depending on scientists’ previous achievements and research area. In their study of Italian scientists, Maisano et al. (2020) demonstrated that grant recipients’ research performance did not significantly differ from that of researchers who did not receive a grant, at least in the short term. According to the data on the Turkish foundation, grant funding did not lead to a significant uptick in publications and their subsequent citations (Tonta, 2018). Elsewhere in Europe, a study on the Russian Presidential Grants did not indicate an impact of funding on grant recipients’ research performance in medicine (Saygitov, 2014). A similar lack of impact could be found when comparing approved and rejected applicants for the German Research Foundation (Hornbostel et al., 2009) and the United States National Institutes of Health (Jacob and Lefgren, 2011). Even when an effect has been identified, it has often been relatively small (Morrillo, 2019; Wang & Shapira, 2015).

The wide variance in research results may be due to the peculiarities of specific foundations, which differ in their selection procedures and general policies. National public funders often do not only target citations and publications, but pursue other strategic objectives (Gök et al., 2016). For example, a comparative study found that while Chinese foundation grants tended to result in a high number of publications, European Union grants tended to be less effective, which could be explained by their emphasis on projects’ social impact (Wang et al., 2020). Besides, in a context where the foundation is the principal funding source, the effect on research performance tends to be much stronger in comparison to countries with less concentrated grant competitions. Gyorffy et al. (2020) pointed to a 47% increase in research output following the receipt of a basic foundation grant, which could be attributed to significantly less funding being available from other sources for unsuccessful applicants.

In terms of size, large grants have significant transaction costs and, therefore, tend not to be as effective as one might expect (Campbell et al., 2010; Clark & Llorens, 2012). A substantial number of empirical studies have indicated that resource distribution across smaller grants has yielded better performance on average than distribution into fewer and larger grants (Aagaard et al., 2019; Mongeon et al., 2016). Based on empirical research on the Center of Excellence Program in four Nordic countries, larger grants appeared to exert a limited impact for already prolific and distinguished groups, vis-à-vis less recognized groups with few other funding options (Langfeldt et al., 2015). Such findings seemed to suggest that under a policy objective of maximizing research output, giving smaller grants to more researchers represented a better course of action.

Another empirical strategy has been to focus on data at the country level. Research results regarding domestic and international grants for several European countries revealed a link between a combination of diverse grants, especially in complex EU programs, and the emergence of highly cited papers (Gök et al., 2016). For small countries in Eastern Europe, funding appeared to be a significant predictor of impact on citations (Gök et al., 2016). This effect could also be achieved through international collaborative research, where funding is a complementary mechanism (Yan et al., 2018; Zhao et al., 2018).

The efficiency of grant funding may be particularly dependent on the research field (Morillo, 2019). Resource distribution itself is not equal across different fields of science: European foundations have typically offered fewer opportunities to support SSH projects than those within the natural sciences and engineering (Grimpe, 2012). Research area has also turned out to be a significant factor in funded projects’ publication performance (Yan et al., 2018). Data on public funding in South Africa revealed an associated increase in publications in the natural sciences, while in engineering and the social sciences, the number of publications and citations did not appear to grow vis-à-vis the control group (Fedderke & Goldschmidt, 2015). The limited impact of grant funding may be further explained by how the process of grant evaluation has been organized to allocate funding to the most productive researchers (Technopolis, 2008). There are at least two reasons why grants may not be distributed among the most productive scientists, as explored below.

First, the most prolific authors may choose not to apply for grants, on different grounds. Grimpe (2012) demonstrated that even the most prominent competitions failed to attract the best scientists in Germany. Top-level scientists have been found to be less inclined to change their research plans to fit the foundations’ agendas (Laudel, 2006). In Italy, a study by Bertoni et al. (2021) revealed that while many of the country’s less productive researchers applied for a grant program, many of the highly productive ones did not. The latter authors suggested that researchers tended to misestimate their position in productivity distribution in relation to the assigned threshold, leading to the most productive not applying for funding.

Second, allocation procedures may not be meritocratic, resulting in a failure to reward the best scientists. According to several studies, the review process has rarely provided reliable results in predicting future scientific productivity (Fang et al., 2016; Graves et al., 2011; Gyorffy et al., 2018). For example, Gyorffy et al. (2020) indicated that grant application reviewers’ scores were only minimally better than random parameters. They found that the best predictors of future research performance were rather the bibliometric indicators of the project leaders at proposal submission. Hence, studies have provided evidence for the promotion of metrics and the demotion of review panels in selecting grant recipients. Researchers have explicitly suggested that “the proposal evaluation process could be more evidence-based and shortened through the more intensive usage of past publication data” (Gyorffy et al., 2020). In this regard, it can be argued that the concentration of resources in the hands of highly productive scientists is more likely to provide a high return on investments in science (Maisano et al., 2020).

Due to publishing norms, bibliometrics is not appropriate for every field of research (Van Raan, 1998). Scientists have generally agreed, although with some reservations, that the use of metrics would benefit evaluation purposes in the natural sciences, but that for the social sciences and especially the humanities, they should be applied with discretion (Abramo et al., 2013; Batista et al., 2006; Hicks et al., 2004; Nederhof, 2006). Compared to the basic natural sciences, more SSH output has tended to be published in journals and as monographs in the national language, significantly limiting the number of potential citations (Mosbah-Natanson & Gingras, 2014). Accordingly, our study acknowledges that scientometric indicators for assessing scientific achievements can have different effects in the social and natural sciences. Therefore, we compare a group of project leaders that received grants in 2014 and 2017 years in two areas of science separately: physics and the social sciences and humanities (SSH).

We propose that increased reliance on international databases would affect different social and natural sciences given their different experience in publishing internationally. Bibliometric analysis shows that natural sciences dominate the publications from post-Soviet countries that appeared in indexed journals; in Russia, it accounts for 78% of all Scopus-indexed papers (Chankseliani et al., 2021). The proportion of social sciences is less than 3% and has slightly increased during the last decade (Chankseliani et al., 2021). Given the development of the social sciences and humanities rather locally than globally, we expect to find the difference in strategic behavior regarding getting articles published – social sciences and humanities authors would be more inclined to choose a more guaranteed way of publishing in indexed journals.

Starting in 2014, the Russian Science Foundation annually maintains several competitions. We focused on the competition named “Fundamental scientific and exploratory research led by research groups,” which in 2014 was the main competition for Russian scientists – winners of this call accounted for 78% of all RSF’s grants awarded this year. Until 2017, the RSF did not demand the project leaders to publish solely in Scopus or Web of Science, and the year 2017 was the first year when these publications became necessary for an application not only for natural scientists but also for social sciences and humanities. Given that there were only several months between the call of application and the submission, scientists did not have enough time to change their publication behavior specifically to provide publications for the competition. However, it is well-known that between 2014 and 2017, scientists started to publish in journals indexed in international citation databases. In this period, the emphasis on international publications has permutated Russian science policy in different contexts, creating intense pressure for Russian scientists to change their publication behavior.^① By 2017, the number of researchers with papers in indexed journals had increased, and it is noteworthy that only these scientists were eligible to participate in the grant competition. More specifically, we focused on whether the policy change resulted in the selection of project leaders with significant publication records considering that researchers who did not publish in highly competitive international journals but instead chose Russian indexed journals or predatory sources were also eligible to apply.

Our data collection process began with assembling a list of all principal investigators who had received project funding from the Russian Science Foundation (RSF) in 2014 and 2017 (Figure 1 represents the main steps in data processing). Though the results of the RSF competitions were available on the foundation’s website, they were not in machine-readable form, and the list only provided data on the funded projects. Only part of the information was publicly available (i.e., project leaders’ names, project abstracts, and a list of publications as a result of the funded projects). Our analysis included two groups of grantees: 1) physicists and 2) social sciences and humanities (SSH) scholars.

Figure 1.

The next step was to gather data on the listed principal investigators’ publications indexed on Scopus. We selected the Scopus database over Web of Science (WoS) as the former covers more Russian-language journals than the latter (Moed et al., 2018). Sterligov (2017) expressed the view that WoS sold “top quality,” while Scopus sold “top scope,” which could explain why the number of journals indexed on Scopus was almost twice as high as on WoS. Indeed, although the share of Russian papers grew on both databases during 2012–2016, there was a difference in the types of publications. While the coverage of proceedings expanded on both Scopus and WoS, the higher number of publications indexed on Scopus also reflected the inclusion of more Russian-language journals (Moed et al., 2018).

The observational window for our study spans the five years preceding the application year, as specified in the scientometric requirements for grant applicants. As mentioned, our data consisted of 209 grantees in 2014 who published 5,701 indexed papers from 2009 to 2013, and 123 grantees in 2017 who published 3,466 indexed papers from 2012 to 2016. We also obtained information from the list of publications resulting from each funded project located on the foundation’s website (4604 publications for the first wave and 6649 publications for the second wave).^②

For the main analysis, we restricted our sample to journal articles and omitted other categories, such as book chapters and conference reports. This decision stemmed from our empirical strategy, as we intended to gather comparable information on the quality of publications across different research areas. Furthermore, the articles were matched with the bibliometric characteristics of the journals where they had been published. We assigned each journal to its respective quartile within the scientific field, based on the journal’s rank according to CiteScore metrics. If the journal belonged to several categories, then the quartile was taken from the category in which it was the highest. The researchers challenge the assumption that journal metrics are a strong indicator of paper quality (Larivière & Sugimoto, 2019). However, due to the limited data availability, we relied on the quartile-ranking as it was the most easily accessible data for the publications. In this regard, we followed the strategy of Gyorffy et al. (2020), who received reliable results by using quartile-ranking in studying Hungarian researchers.

To gauge research output, we gathered and stored the list of grantees’ publications available on the RSF website (4604 publications for the first wave and 6649 publications for the second wave). This step allowed us to compare principal investigators’ prior output with publications which resulted from the funded projects. We chose to analyze only journal articles at this stage, as it was not possible to manually check each publication through Scopus (5020 articles for the 2014 competition and 2948 articles for the 2017 competition). Similarly to the principal investigators’ publication records, the latter publications were matched with the bibliometric characteristics of journals where articles were published.

This study has several sources for limitations regarding the nature of available data. Available data did not allow the analysis of non-overlapping periods and the equal number of grant-holders in each period, given that the different numbers of grants were awarded in 2014 and 2017. By focusing on later years, we could avoid the overlapping; however, we aimed to analyze the first year when the bibliometric requirements were strengthened. Further, although we compared the results of selection procedures between two periods, the research design differs from the standard before-and-after study, which usually measures outcomes in a group of participants before an intervention, and then again afterward. In this study, the sample for each year consists of a different set of scientists, winners of the 2014 competition, and winners of the 2017 competition, and the number of grant-holders in each period is uneven. To make the comparison possible, we provided both absolute and relative perspectives – raw numbers on the principal investigators’ publications and citations and size-independent indicators, including shares of papers in Q1 journals and an average number of publications per author.

In the modern era, peer review has dominated the evaluation of grant proposals, meaning that reviewers’ scores tend to be crucial in deciding which proposals to fund (Azulay, 2020). However, research from recent years has brought the efficiency of peer review into question by pointing out that it might add zero or even negative value if the reviewers were to be biased, mistaken, or focused on different goals (Guthrie et al., 2018; Gyorffy et al., 2020; Li & Agha, 2014). Empirical research on possible links between peer-review scores and project outcomes has brought mixed results (Li & Agha, 2014; Luukkonen, 2012).

Recent empirical studies have posited that using scientific merit scores in evaluating grant proposals may yield better outcomes (Azulay, 2020; Fedderke & Goldschmidt, 2015; Gyorffy et al., 2020). Based on this evidence, should funders rely more on quantitative metrics like publications and citations in addition to peer review? The RSF case is unique due to the foundation’s formal policy of using publication metrics for the initial threshold in the application process, as it requires a project leader to have published a certain number of indexed papers before applying. To study the effects of tightening this policy, we compared two groups of grant-holders who participated in completions before and after the policy raised the publication threshold for principal investigators. Although scientometric requirements have led some Russian institutions to employ questionable strategies, including publishing in predatory journals (Guskov et al., 2018), our study demonstrates that using metrics results in a more effective selection of project leaders. We showed that in Russia, with its problems with academic expertise, reliance on quantitative indicators yields better results regarding selecting more prolific authors.

First, we found that the selection among physicists in the first period had already been effective, as the grants tended to be allocated to prolific authors publishing in journals of high quality. In the second period, the grants were awarded to researchers with slightly better metrics compared with the first wave of the grant competition. By contrast, the SSH project grantees had been less prolific in publishing internationally; their smaller number of publications in the most selective set of journals could be explained by the field’s traditions and limited journal coverage in international databases. At the same time, between the first and second periods, the selection of SSH grantees favored those researchers who have more impressive publication records, both in terms of the quantity and quality of publications operationalized in this research on the basis of journal quartile.

Second, in physics, proposals submitted by investigators with better metrics tended to yield better publication records during grant realization that could be attributed to the more profound role of the project leader: Project publications in this field are generally strongly shaped and authored by a principal investigator. We found that the correlation between metrics of the principal investigators at proposal submission and project output did not differ between periods. In SSH, limited observations did not allow us to conduct correlational analysis. We assume that in this case the role of the project leader is less evident, as many project publications appeared without their participation as coauthors.

Using metric thresholds could change the supply side of those able to apply for funding. While this is less important for fields with many prolific authors, it is crucial for research fields where fewer authors tend to publish internationally. For SSH, the use of the metrics for initial filtering might lead to a change in the supply side of eligible researchers, as the bibliometric threshold defines what type of scholarship counts as publications through the lens of international citation databases. The exact answer requires the data on unsupported applications; unfortunately, access to them is restricted. Our data provides only indirect confirmation of our suggestion by showing that between 2014 and 2017, the pool of authors who were eligible to apply increased dramatically, and this increase has happened at the expense of quality given that significant (more than 50%) share of papers was published in low-quality journals. In this context, however, the grants were awarded to researchers with better publication metrics.

Our findings indicate that formal policy may have profound effects on the social sciences and humanities, which have been considered less suitable for metric-based evaluations. We observed that the share of non-journal publications among new grantees decreased, while the share of English-language journal articles increased. Our results confirm the findings by Hammarfelt and de Rijcke (2015) that evaluation practices changed publication patterns in the humanities by increasing the most common type of output (i.e., English-language journal articles). We did not focus in detail on changes in publication content. However, like others, we suggested that format changes would require complying with the topics, perspectives, and methods supported by the major journals in a field (Gantman & Fernandez, 2016; Koch & Vanderstraeten, 2019; López Piñeiro & Hicks, 2014). Hence, this international path might lead to the suppression of certain research themes and styles, and to abandoning important roles which social scientists have come to play in their societies (Sokolov, 2019). In this respect, foundations employing broad categories such as “social sciences and humanities” may create a situation where the use of metrics privileges research fields where it is easier to publish internationally (e.g., cognitive psychology versus history).

The evaluation of grant applications is a challenging task for any funding organization. The process consumes human resources, requiring extensive use of experts (Gyorffy et al., 2020). In the case of countries with a good pool of researchers, quantitative metrics might be used for the initial screening, which would enable peer reviewers to focus on distinguishing among top-performing scholars (Asulay, 2020; Mali et al. 2017). In other words, reliance on metrics at an early stage can speed up the evaluation process, allowing for expertise to be used where it is most needed, “without wasting resources for proposals that are highly likely to be accepted because of their authors’ recent publication activities as well as for proposals that are unlikely to be accepted due to extremely weak prior publication performance” (Gyorffy et al., 2020).

The application of bibliometric indicators is more promising and not limited to their capacity to lessen the administrative burden related to the evaluation of grant applications. What is more important is that the use of bibliometric indicators can counteract issues with objectivity and fairness in a peer review. Evidence suggests that even in contexts with strong scientific ethos, such negative social phenomena as nepotism is present (Bornmann, 2011). Sandström and Hällsten (2008) show that grant-holders with reviewer affiliation receive a higher score than applicants without affiliation. Van den Besselaar (2012) demonstrated that being part of a council’s inner circle benefits the number of received grants compared to applicants more at a distance.

In countries where academic expertise is corrupted, the objectivity of decisions in review panels is a much more intense issue. The post-soviet grant system lacks transparency in the selection process and has the potential for corruption (Batygin, 2001). Peer review is only effective when there is a shortage of competent academics, and the evaluation results are trusted. However, expert status can be obtained without confirmation from significant scientific achievements, thus making expert evaluations less legitimate. Severe flaws in academic integrity in Russia have been widely acknowledged and discussed throughout the scientific community, by state officials, and the general public. In Russia, even an academic degree, the primary signal of academic quality, may fail to be reliable proof of academic achievement (Sokolov, 2021).

In this context, bibliometric indicators might be considered as an instrument that contributes^③ to “the fairness of research evaluations by presenting ‘objective’ information to a peer review that would otherwise depend more on the personal views and experiences of the scientists appointed as referees” (Južnič et al., 2010: 431). Metrics are perceived as trustworthy because they result from the review process of scholarly journals and publishers (Langfeldt et al., 2021). The editors and reviewers of the international journals are external experts, highly competent compared with local academics, and free from incentives to influence the results of grant allocation. Južnič et al. (2010) presented evidence of how bibliometric indicators in the absence of a sound international peer review could prevent reviewers from evaluating former students or colleagues favorably if their publication and citation record would indicate their underperformance compared with other applicants.

It is a highly discussable question whether the use of bibliometric indicators can counteract concerns in a peer review related to its objectivity and predictability of significant research results. While the survey of reviewers showed that they had already been using metrics in evaluating grant proposals (Langfeldt et al., 2021), our knowledge of the possible effects of such policy has been insufficient, as foundations have rarely explicitly relied on metrics in the proposal evaluation process (Mali et al., 2017). Developing this practice into a formal policy would make it more legitimate, allowing for elaboration of rules on how to properly employ metrics and for a chance to gather additional information on policy effects.