The Flemish Performance-based Research Funding System: A Unique Variant of the Norwegian Model

Performance-based research funding systems (PRFSs) have been installed in several countries around the globe, many of which are European (Debackere et al., 2018; Zacharewicz et al., 2018). Hicks (2012) has characterized PRFSs as “national systems of research output evaluation used to distribute research funding to universities”. The Flemish Government introduced the BOF-key (BOF stands for “Bijzonder Onderzoeksfonds” or “University Research Fund” in English), a mechanism to distribute research funding between the Flemish universities, in 1994 (Spruyt & Engels, 2013). Originally the BOF-key was based primarily on input indicators. A major change occurred in 2003 with the addition of bibliometric parameters based on publication and citation data from the Science Citation Index Expanded (SCIE) of Web of Science (WoS) to the system (Debackere & Glänzel, 2004). From then onwards, the BOF-key became a PRFS. This change, however, triggered strong criticism among the scholarly community in Flanders (De Wever, 2007), as the nature of the SCIE caused scholarly publications in the social sciences and humanities (SSH) to be of negligible importance in the BOF-key (Verleysen, Ghesquière, & Engels, 2014). This spurred on two further reforms in 2008, namely (1) the inclusion in the bibliometric parameters of publications and citations indexed in the Social Sciences Citation Index (SSCI), the Arts and Humanities Citation Index (AHCI) and the WoS conference proceedings databases, and (2) the enactment of the legal framework for a comprehensive coverage database of scholarly peer reviewed publications in the SSH, the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW; Engels et al., 2009). The data collected in the VABB-SHW resulted in the addition of a new bibliometric parameter to the BOF-key as of 2011 (Engels, Ossenblok, & Spruyt, 2012). After the first 5-yearly external evaluation of the VABB-SHW (Zuijdam et al., 2013), the government decided to gradually increase the weight of the bibliometric parameters in the BOF-key to 16.6% for WoS-indexed publications, 6.8% for VABB-SHW indexed publications and 16.6% for WoS-indexed citations of WoS-indexed publications as of 2016 (Spruyt & Engels, 2013). Since 2016, the BOF-key consists of the following parameters:

Overall, the Flemish PRFS has been well documented, e.g. in the aforementioned publications. The introduction of bibliometric parameters in the BOF-key followed a tradition of studying the possibilities, limitations and implications of the use of bibliometrics as a science policy tool (e.g. Luwel, Noyons, & Moed, 1999; Moed et al., 1998; Van Den Berghe et al., 1998) including in the humanities (Billiet et al., 2004; Moed, Luwel, & Nederhof, 2002) and in law (Luwel et al., 1999). Recently Verleysen and Rousseau (2017) have also discussed to what extent the bibliometric parameters of the BOF-key can help evaluators to act according to the principles of the Leiden Manifesto. They conclude that, although further improvements are possible, the changes that have been made over the last decade have resulted in ever greater compliance with the 10 principles of the Leiden Manifesto. Complementing the aforementioned references, the focus of this contribution is on the history of the Flemish PRFS, as well as the inter-institutional organizational setup and processes that underpin the bibliometric parameters of the Flemish PRFS.

The Flemish universities played a significant role in the development of the Flemish PRFS since the early 1990s. Initially the Flemish government encouraged the five Flemish universities (i.e. Antwerp University, Ghent University, Hasselt University, KU Leuven, and Vrije Universiteit Brussel) to experiment with research evaluation underpinned by bibliometrics. In 2001 the government decided to fund a group of researchers at KU Leuven with, among other things, the task to develop and implement a bibliometric indicator for inclusion in the BOF-key. In 2007 all Flemish universities became part of this consortium, which became known as the Flemish Center for R&D Monitoring (ECOOM) from 2009 onwards (te Velde et al., 2018). ECOOM is financed through a five year consortium agreement with the government, stipulating among other things the expectations with regard to the delivery of the bibliometric indicators for the BOF-key. The ECOOM-Leuven bibliometrics group headed by Wolfgang Glänzel is responsible for the WoS-based bibliometric indicators, whereas our own ECOOM-Antwerp group has the task to annually deliver the VABB-SHW. For more information on the tasks of the other ECOOM groups we refer the reader to www.ecoom.b All members of ECOOM report to its steering committee, which is chaired by a representative of the Flemish Minister for Innovation. At the Flemish level, several working groups operate on behalf of ECOOM, including the interuniversity VABB-SHW working group chaired by Koenraad Debackere, who is ECOOM’s principal investigator.

The BOF-regulation enacted by the Flemish government stipulates the criteria and rules according to which the BOF-key, including its bibliometric parameters, need to be calculated. In addition, the government has established the GP (“Gezaghebbende Panel” or Authoritative Panel), which is composed of 18 professors affiliated with Flemish universities and whose expertise covers the main SSH disciplines. In view of the annual new version of the VABB-SHW, the GP is entrusted by the government with the task of selecting the publication channels which comply with the criteria set in BOF-regulation ((Verleysen, Ghesquière, & Engels, 2014). In addition, the GP can apply additional criteria and propose changes to the BOF-regulation to the government. The members of the GP are appointed by the government and hence as a group report to the government only. Administrative support to the GP is provided by the VLIR (“Vlaamse Interuniversitaire Raad” or Flemish Rectors Conference), whereas ECOOM-Antwerp provides technical support. This technical support consist of, among other things, providing the GP with overviews (both overall and per discipline) of publication channels that need to be decided upon, as well as bibliometric analyses that can inform their decision-making processes (e.g. in relation to predatory publishing).

For the WoS-based bibliometric indicators of the BOF-key the government decided not to install any specific working groups or panels. However, in view of the consecutive changes to the BOF-key and the WoS-publications parameter in particular, the research policy working group of the VLIR has taken up an active role in discussing ideas, limitations and possibilities in order to arrive at common proposals to the government (Spruyt & Engels, 2013). Indeed, the government has also repeatedly encouraged the universities to come forward with consensus suggestions to improve the BOF-key and the BOF-regulation. As such, the BOF-key should not be seen as something forced on the universities by the government. Instead, the discussions of the BOF-key over the years have allowed the universities to develop considerable expertise with regard to PRFS, leading to full recognition of pros and cons. It therefore does not come as a surprise that a recent evaluation of the BOF-key and BOF-regulation resulted in only minor suggestions for change. Moreover, the international panel of experts observed that the year-to-year changes in the shares per university are rather small (Korlaar, Bongers, & Groot Beumer, 2018).

The data sources for the bibliometric indicators of the Flemish BOF-key are the WoS and the VABB-SHW. For all three bibliometric indicators, a 10 year time window is used.

The BOF-money as distributed through the BOF-key is essential for each of the Flemish universities in view of their research mission. In 2017 the BOF-money amounted to 170.8 million Euro, representing 8.9% of the total turnover of the Flemish universities (Korlaar et al., 2018). As such, the BOF-money has a structural impact on the research capacity of each of the Flemish universities. However, the direct funding implications of the bibliometric parameters, which together amount to 40% of the BOF-key, have been limited in recent years. One reason is that the BOF-regulation guarantees a minimum share for the three small and medium sized universities (SMUs) in Flanders, because of the importance of a regionally diverse university landscape. At the same time, this has may have led to increased competition between the two largest universities. Another reason is that, since the increase of the weight of the VABB-SHW parameter in 2013, the relative weight of each of the bibliometric parameters has been stable. Hence after the initial substantial impact of the addition of the VABB-SHW (Zuijdam et al., 2013), the BOF-key has stabilized since 2013.

The data collected for calculation of the BOF-key are also used for monitoring purposes. In particular, the annual Flemish Indicator Book contains sections on publication output and impact (Debackere, Glänzel, & Thijs, 2017; Guns & Engels, 2017). The extent to which the universities use the data for internal allocation or evaluation purposes is unknown. Already upon the introduction of the bibliometric parameters in the BOF-key, Debackere and Glänzel (2004) warned against the intra-institutional use of the data in view of allocation and evaluation. Nevertheless, successive evaluation panels have observed that such use does occur (Korlaar et al., 2018; Zuijdam et al., 2013), notwithstanding the clear opposition of the GP against such use. The data are also used as a benchmark when researchers apply to become panel members at the Research Foundation Flanders (FWO, Fonds Wetenschappelijk Onderzoek Vlaanderen). In almost all cases, however, such use of the data is informing rather than replacing peer review.

The data collected for the VABB-SHW have also been used to study publication patterns in the SSH. Engels, Ossenblok and Spruyt (2012) studied changes in publication patterns in the SSH using VABB-SHW data. Among other results, they find that the number and share of English-language publications is increasing, to the detriment of Dutch (the local language) and other languages. The use of different publication types differs from discipline to discipline; on the whole, the share of book publications in the humanities is higher and, contrary to the social sciences, not decreasing. In a separate study (Ossenblok, Verleysen, & Engels, 2014), it was found that patterns of coauthorship can also be very different among disciplines. Follow-up studies have focused on the role of editors of edited books in collaboration and the prevalence of alphabetical publishing (Guns, 2016; Ossenblok & Engels, 2015). The VABB-SHW has also allowed for several studies of the characteristics of author types that have hitherto remained largely unstudied, such as monograph authors or book editors (Ossenblok, Guns, & Thelwall, 2015; (Verleysen & Ossenblok, 2017). In two studies, Verleysen and Weeren (2016a, b) clustered senior authors in the VABB-SHW by their publication patterns. The results show that most disciplines are heterogeneous in terms of publication patterns of the authors. Given the importance of book publications in the SSH, several studies have been undertaken that investigate the characteristics of book publications and publishers: internationalization of book publishing (Verleysen & Engels, 2014a, 2014b), the role of book publications in History (Verleysen & Engels, 2012), and concentration of academic book publishers (Guns, 2018). Finally, given the prevalence of similar national databases across Europe (Sīle et al,, 2017; Sīle et al., 2018), many possibilities have opened in the last few years for international comparison (Engels et al., 2018; Giménez-Toledo et al., 2016; Kulczycki, Engels, & Nowotniak, 2017; Kulczycki et al., 2018; Ossenblok, Engels, & Sivertsen, 2012; Pölönen et al., 2018; Pölönen et al., 2017; Pölönen et al., 2017). ECOOM-Antwerp has also setup an online overview of national bibliographic databases for the SSH at https://ecoom.uantwerpen.be/sshdatabases

While the Flemish PRFS has developed independently, some of the most substantial changes, especially those relating to the VABB-SHW, have been strongly inspired by the Norwegian model. Specifically, the comprehensive coverage of the VABB-SHW database as well as the different weights assigned to different publication types in the VABB-SHW are similar to the Norwegian system. What is different from the Norwegian model, however, is the split of publications over two databases—Web of Science and VABB-SHW—as well as the use of WoS citations as a parameter. Another difference is that publication channels in the VABB-SHW are not weighted by quality level: either a channel is included or it is not. Journal articles that are indexed in WoS are weighted by the JIF rather than a predefined set of quality levels established by panels. Furthermore, the Flemish PRFS uses whole counting instead of the fractional counting one encounters in the original Norwegian model (Sivertsen, 2016).

It is well understood that PRFSs can have an effect on the publishing characteristics in a country (Hicks, 2013). For the case of Flanders, this has been empirically studied by Guns and Engels (2016). While causal effects are impossible to establish, they find that a strong emphasis on WoS-indexed publications since 2003 has been accompanied by a growth in WoS publications that is greater than what can be observed in other countries. However, during the same period the number of researchers in Flanders also increased rapidly (Delanote et al., 2017). Remarkably, the introduction of the VABB-SHW has not led to slower growth of WoS publications in the SSH. Publication types that only count in the VABB-SHW parameter of the BOF-key appear to follow a different mechanism; especially the growth in yearly number of book chapters is comparable to that of WoS articles.

One particularly interesting effect to study is that of the whole counting of publications per university. Debackere and Glänzel (2004) warned that any systematic collaboration between institutions could distort the bibliometric parameters of the BOF-key and hence make it unacceptable. Sivertsen (2017) also suggested that counting methods should incentivize collaboration without stimulating the inclusion of authors with minimal contributions. Indeed, the difference in counting method between the Norwegian model as implemented in Norway and the counting method in Flanders is striking, calling for a systematic analysis of the possible differential effects of both.

The overall effects of the Flemish PRFS range much wider, however.

Since 2004 the bibliometric parameters discussed in this article are also used in another interuniversity allocation scheme, namely the IOF-key (Industrial Research Fund). The IOF-money, which amounted to 32 million in 2018 and is thus much less than the BOF-money, stimulates the universities to develop new technologies and prepare them for use in the market place. In addition to bibliometric indicators, the IOF-key also takes into account PhDs defended, industrial research funding and EU-framework program funding, license income, patents, and spin-off creation. As such, the BOF-key became a gateway for the introduction of performance-based funding also in the area of valorization and development.

In 2008, the government went one step further by taking the bibliometric parameters of the BOF-key also into account in the calculation of the government block grant. This expansion of the reach of the bibliometric indicators was not without controversy and still regularly causes debate. One often voiced concern is that the balance between the education and the research mission of the university has tilted too much towards the research aspect. Inversely, one could also argue that more emphasis on research was needed, at least at the time when the bibliometric parameters were introduced into the BOF-key. It seems impossible, however, to pinpoint such arguments in one or the other direction.

There seems to be a broad consensus, however, that the research performed at the Flemish universities is of high quality, and that this is partially thanks to a close monitoring of research activity and impact. Indeed, the datasets that have been set up in view of the bibliometric parameters of the BOF-key have become references for many other processes at the level of institutions and the government. Still, the conceptual definition and delimitation of research publications remains challenging given the constant changes in scholarly communication practices. In the current era of digital and open science, these definitions and delimitations will also need to co-evolve with the broader evolutions of the ways in which research is taking place.

Among researchers such overall benefits of the PRFS are not generally shared. One argument is that the PRFS has led to undue pressure to publish. This point of view is, for instance, regularly expressed in opinion pieces in the Flemish press, yet it is hard to directly ascertain to what extent such a trickle-down effect takes place. A first consideration is that almost all researchers operate in the international context of their discipline. The need and wish to contribute to the development of the field generates pressure for all researchers who are at the forefront of their field. The degree to which the epistemic cultures of the researchers are in line with the workings of the PRFS may differ from specialism to specialism, thus causing inconsistency of expectations more for some than for others. The weighting of WoS-publications, for example, may be more in line with the hierarchical structure of the STEM journal landscape than with the publishing landscape of the SSH (Bonaccorsi, 2018). The perceived unintended consequences of the PRFS have been stressed in particular in relation to violations of research integrity: if too much pressure is put on researchers, they might lose their integrity. Although this is a general and international concern, some have pinpointed blame in particular on the Flemish PRFS (Cornelis, 2013). To the best of our knowledge, no evidence for this assertion has been produced, and cases of infringement of research integrity have not been exceptionally common in Flanders.

Still, the PRFS might be one of several elements in the evolution of the university into an environment that exerts many short term expectations on researchers. A survey among PhD students in Flanders found that many of them suffer undue levels of stress (Levecque et al., 2017). Also among postdocs and professors, high levels of stress and chronic stress seem to be common. One reason might be that the PRFS is part of a much broader evolution towards more measured and monitored environments (Power, 1999). Dahler-Larsen (2017) has also pointed out that PRFSs may cause ambiguity in terms of expectations for researchers. So while publications become closely monitored and measured, in some cases it may not be clear what to expect in terms of ‘credits’ from a certain publication. Researchers may also disagree with the credits that are assigned, which may cause distress in itself.

From a government perspective, all these issues are important. Above all, however, the government expects investment in universities to translate into economic and societal impact. According to a recent analysis, every Euro that the government invests in Flemish universities generates 6 Euros of wealth in Flanders (BiGGAR Economics, 2017). This occurs through education, research, and knowledge and technology transfer. While the competition for universities is international (in particular for research), the PRFS compares universities within the region. Even though whole counting is used for all parameters—and collaboration is hence encouraged, the fact that the reference for the funding distribution is regional often reinforces the idea that the PRFS is not encouraging collaboration. When the Flemish Minister of Innovation Philippe Muyters announced on the 26^th of September 2018 extra investments in research and innovation, including 35 million Euros of additional BOF-funding, it therefore came as no surprise that he stressed the importance of collaboration, both among universities and internationally. In addition, he invited the universities to come forward with suggestions on how to encourage more interdisciplinarity and excellence through the BOF-means. At the time of writing, no decision has been taken on whether or how to fine-tune the BOF-key in response to these expectations. One possibility appears to be more emphasis on international benchmarking and/or allocation of funding according to goals and strategies rather than measured (by definition: past) performance. It is an open question whether such an approach would in the end lead to more economic and societal impact. Such unknowns notwithstanding, the impact of the Flemish universities is high already.