A Local Adaptation in an Output-Based Research Support Scheme (OBRSS) at University College Dublin

University College Dublin (UCD), Ireland’s largest university, has implemented the Output-Based Research Support Scheme (hereafter “OBRSS”) since 2016. Adapted from the Norwegian model, the OBRSS awards individual academic staff using a points system based on the number of publications and doctoral students. Hence, a major difference between the Norwegian model and the OBRSS is that the Norwegian model is designed to allocate block grants to universities (Schneider, 2009; Sivertsen, 2016), whereas the OBRSS aims to reward academic staff individually. It should also be noted that the OBRSS is implemented as a university initiative rather than a component of performance-based system as described in Hicks (2012) and Zacharewicz, Lepori, Reale and Jonkers (2018).

In this article, we will first describe the design and implementation processes of the OBRSS, including the creation of the ranked publication list and points system, as well as infrastructure requirements. Then, some results of the OBRSS will be presented, focusing on the coverage of publications reported in the OBRSS ranked publication list and Scopus.

Some data about spending will also be reported. Last, challenges such as the evaluation of the OBRSS in terms of fairness, transparency, and effectiveness will be discussed.

The overarching objective of the OBRSS is to incentivise academic staff to publish research output in higher quality outlets. The principles of the OBRSS, defined at the outset, are as follows:

The design involved the construction of a ranked publication list and a points system. The ranked publication list includes journals, conferences, and monographs; and the ranking is based on a basket of indicators including Norwegian Register for Scientific Journals, Series and Publishers (NSD—National Centre for Research Data, 2018), Danish BFI (Ministry of Higher Education and Science, 2018), Finnish Publication Forum (Federation of Finnish Learned Societies), SNIP (Source Normalised Impact Factor per Paper), and CiteScore. Academic staff from across the university are consulted in finalising the ranked publication list each year; they are also requested to update their publication records on the Current Research Information System (CRIS) for points to be calculated. Only publications with a status of 'Published' in the CRIS are included in the OBRSS. PhD supervision records are maintained in the institutional Student Information System (SIS).

The OBRSS uses the ranked publication list—one section for Publishers and another for Series (Journals, Book Series, and Conference Series)—as a reference for the calculation of points. Each publication is assigned one of two levels: level 1–Normal or level 2–Prestigious. Weighted scores are then applied to each publication. Similar to the Norwegian model, points are allocated for different types of publication as summarised in Table 1

Figure 1

There is a consultation process to ensure that inputs from the academic staff are considered in finalising the ranked publication list. During the consultation period, academic staff can make recommendations to add/remove publications to/from the ranked publication list at the two levels. The suggestions and recommendations are reviewed by the Office of Research Administration. Considering the objectives and scope of the OBRSS, external panels are not used to review the ranked publication list.

Publication points are calculated for each academic staff’s publications in the CRIS over a three-year period (for example 2015–2017) using the following formula (Table 2):

The total publication points for an individual are equal to the sum of all the points for each of their publications in the three-year period. PhD supervision points are calculated by counting the number of PhD students supervised by each academic staff member in the current academic year. Two points are awarded for being a primary or a secondary supervisor. The maximum points for PhD supervision have been capped at 20. Publication and PhD supervision points are both worth €35.

All academic staff are automatically entered into the OBRSS each year. The total points that an academic staff has accumulated is communicated using a personalised points statement. Final points statements are issued to academic staff receiving an award in October each year. The minimum value threshold for a research award is €200. There is no maximum research award, but in the first two years of operation, the maximum award based on the maximum points for an individual author were between €10,000 and €15,000.

Awards may be used by the academic staff for research support, such as to cover travel expenses, office supplies, equipment, and laboratory supplies. Overall, approximately 1% of the total annual research budget for the university is allocated to the OBRSS.

The first two years of the OBRSS has provided some valuable data for understanding research activities in the university. The more complete record of publications is essential and helpful for deliberating research strategies, on the one hand, and the spending pattern gives insights into the type of activities and resources that academic staff consider important for supporting their research, on the other. The following sections present a comparison of the coverage of publications in Scopus and the OBRSS ranked publication list, changes of publications reported and of the ranked publications list in the first two years, as well as some data about spending so far.

3.1

Coverage of Scopus and OBRSS

One of the most significant outcomes of the implementation of the OBRSS is a more complete picture of publication records in University College Dublin. The number of academic staff updating their research profiles in the CRIS has increased each year. In the first year the OBRSS was implemented, 85% of academic staff updated their profiles as opposed to 75% over the previous three years.

Using the publication records in CRIS, we can compare the coverage of research outputs per School and College in Scopus and the OBRSS ranked publication list. Although many international university ranking organisations use either Scopus or Web of Science as a data source to evaluate the research performance of an institution, the data source only works well for STEM (Science, Technology, Engineering & Mathematics) disciplines where coverage can be as high as 94% (Physics). For Arts & Humanities disciplines, coverage of their outputs varies between 2% (Irish, Celtic Studies and Folklore) and 18% (English, Drama & Film).

Table 3

Comparison of publications for academic staff only from 2013 to 2017 inclusive; Scopus data from SciVal 25 May 2018; CRIS data from UCD RMS Profiles 22 June 2018.

UCD School Name (Discipline)	Scopus Total 2013–2017	CRIS Total 2013–2017	% Coverage in Scopus
Agriculture & Food Science	868	1,085	80.0%
Archaeology	73	202	36.1%
Architecture, Planning and Environmental Policy	127	580	21.9%
Art History & Cultural Policy	12	141	8.5%
Biology & Environmental Science	448	572	78.3%
Biomolecular & Biomedical Science	468	522	89.7%
Biosystems and Food Engineering	470	665	70.7%
Business	459	1,047	43.8%
Chemical & Bioprocess Engineering	250	266	94.0%
Chemistry	391	452	86.5%
Civil Engineering	205	470	43.6%
Classics	5	53	9.4%
Computer Science	767	916	83.7%
Earth Sciences	149	390	38.2%
Economics	147	186	79.0%
Education	83	178	46.6%
Electrical & Electronic Engineering	730	860	84.9%
English, Drama & Film	78	415	18.8%
Geography	86	324	26.5%
History	40	291	13.7%
Information & Communication Studies	61	149	40.9%
Irish, Celtic Studies and Folklore	3	145	2.1%
Languages, Cultures and Linguistics	57	367	15.5%
Law	52	495	10.5%
Mathematics & Statistics	460	617	74.6%
Mechanical & Materials Engineering	478	919	52.0%
Medicine	1,867	2,451	76.2%
Music	7	126	5.6%
Nursing, Midwifery & Health Systems	241	534	45.1%
Philosophy	98	246	39.8%
Physics	1,325	1,403	94.4%
Politics & International Relations	124	321	38.6%
Psychology	277	622	44.5%
Public Health, Physiotherapy and Sports Science	747	977	76.5%
Social Policy, Social Work and Social Justice	128	437	29.3%
Sociology	46	267	17.2%
Veterinary Medicine	626	1,094	57.2%
Grand Total	12,453	20,785	59.9%

In total the UCD CRIS system records approximately 4,000 publications records per year for academic staff as opposed to Scopus which records 2,500 per year, see Table 4 below:

Table 4

Comparison of publications for academic staff only from 2013 to 2017 inclusive; Scopus data from SciVal 25 May 2018; CRIS data from UCD RMS Profiles 22 June 2018.

UCD College Name	Scopus Total 2013–2017	CRIS Total 2013–2017	% Coverage in Scopus
College of Arts & Humanities	228	1,556	14.7%
College of Business	468	1,046	44.7%
College of Engineering & Architecture	2,384	3,691	64.6%
College of Health and Agricultural Sciences	4,017	6,021	66.7%
College of Science	3,925	5,075	77.3%
College of Social Sciences & Law	1,227	3,487	35.2%
Grand Total	12,249	20,876	58.7%

The distribution of publication output in the OBRSS shows that the coverage is consistent across Colleges, apart from the College of Science. The anomaly (Figure 2) is due to large volumes of papers produced in the School of Physics through international collaborations. It is possible to have up to 5,000 authors on single papers in Physics (Castelvecchi, 2015), leading to a small number of academic staff in the School of Physics producing a large volume of publications in both Prestigious Level 2 and Normal Level 1 channels.

Figure 2

3.3

Research Funding and Spending

Since the implementation in 2016, over €1.3m in new research funding was allocated to academic staff to support their research activities. The number of recipients and average award value have both increased from 2016 to 2017.

Figure 3

Interestingly, over 50% of the awardees have not spent the research support fund at all. Of those who have used their funds, the expenses have been claimed for travel expenses, office supplies, equipment, and laboratory supplies:

41% of the funding was used to cover travels inside and outside EU. This included accommodation, transport and subsistence expenses;

25% of the funds spent were used to buy office supplies. Examples of the items purchased are: books, subscription to journals, staff training courses, website designs and copy-printing;

11% of the funds spent were used to buy equipment, such as PC’s, laptops, peripherals and laboratory or office furniture;

8% of the funds spent were used to purchase laboratory supplies, such as chemicals, parts, disposables, glassware & plastics and other general supplies.

Implementing the OBRSS requires sound infrastructure, including the Current Research Information Systems, that supports reporting of publication coverage and trends, as well as spending pattern. Substantial resources were also needed to create and maintain the ranked publication list. In fact, the construction of the ranked publication list and points system are the basic steps to make OBRSS work. When compiling the publication list, suggestions and recommendations from academic staff are essential to gauge the completeness of the list as well as the appropriateness of the rank assigned. However, sometimes disciplinary differences are difficult to reconcile. For example, a publisher could be considered as prestigious in one discipline while it is considered normal in another. Some suggestions also fall into specialised areas where the publication is not indexed in Scopus and is not included in the Danish, Finnish, or Norwegian list. The decision about inclusion or exclusion of the publication could be taxing when balancing the credibility and fairness of the list. Nevertheless, as the ranked publication list is being updated every year, it is expected that the scope and the ranking will be adjusted to reflect quality, impact, and disciplinary norms.

Another challenge is to evaluate the effectiveness of the OBRSS pertaining to the objective of increasing publications in high quality outlets. The main reason is that many factors can contribute to different publication trends, for example, research areas of new staff members, national and EU funding priorities, and so on. The effectiveness of the OBRSS would be inconclusive despite it would certainly be a contributive factor in steering research outputs. Also, there is a risk that the OBRSS could result in a higher number of publications at the Normal Level 1 than the Prestigious Level 2, as Butler (2003, 2004) suggests in her studies of Australia. However, there is evidence from Norway that Level 2 publishing may increase as well (Schneider et al., 2015), which is just as likely since the amount of the award at UCD is rather small. The fact that less than 50% of the awardees has spent the funds is an indication that the award does not provide a strong enough incentive to game the system. It is expected that the ranked publication list would be seen as a guide of high quality publication channels and would alter preferences of outlets accordingly, whilst whether academic staff would be extrinsically motivated would need further investigation.

The OBRSS can also be used in ways unintended by the objectives of the scheme. As of now, the heads of school are provided with the points statement of academic staff in their unit. It is not clear as to whether and how the incentives might trickle down (Aagaard, 2015) or how the scheme might influence perception of university management and policy (Liefner, 2003; Woelert & Yates, 2015). There have been reports that the points have been used for self-evaluation as well as comparison by academic staff and heads of school. The “constitutive effects” (Dahler-Larsen, 2014) would also demand further investigation in the future.

This paper summarises the first two years of implementation of the OBRSS and some data about the coverage of publications and spending pattern. Whilst there have been questions concerning the fairness, transparency, and effectiveness and efficiency of the scheme, the OBRSS has also received encouraging and positive feedbacks. Since most funding schemes, both within the university and those offered by national and EU funding agencies, are competitive, faculty are appreciative of the reward of discretionary funds. The less positive responses of the scheme are largely dissatisfaction or disagreement with the ranked publication list. With the consultation process in place, it is hoped that the list will be updated to reflect quality, impact, and disciplinary norms. Research outputs and publications will be analysed regularly, with the understanding that the OBRSS factor would not be entirely conclusive.

It should also be noted that the points system of the OBRSS is not intended to be used as a tool, not least the sole criterion, for research assessment. Since the OBRSS points system does not represent the impact and quality of all kinds of research output, it does not necessarily reflect individuals’ research performance, particularly for those whose research outputs are more tailored and useful to local audience such as policy makers and businesses. An analysis of research outputs not included in the OBRSS ranked publication list would provide some insights into the notion of impact other than publications.

Nevertheless, the OBRSS has set an example of output-based support scheme in Ireland. Two other universities are currently considering implementing similar schemes. It is not yet known, however, whether they will adapt the Norwegian model when constructing the publication list and the points system.

Based on the experiences in the adaption of the Norwegian model (Aagaard, Bloch, & Schneider, 2015), it will be a few years before publication trends and effects of the OBRSS can be identified and analysed. Nevertheless, the implementation has provided insight into current publication patterns and preferences.