The number of scholarly papers and citations thereto are indirect indicators of the level of scientific development of countries. The number of scholarly papers with Turkish affiliations listed in citation indexes has increased tremendously over the years and Turkey ranks 18th in the world in terms of number of publications. Over 36,000 papers were published in 2015 alone, although their scientific impact in terms of the number of citations they gather is well below the average of the world, the European Union (EU) and the OECD countries.
In 1993, the Turkish Scientific and Technological Research Council (TÜBİTAK) initiated a monetary support program (UBYT) to incentivize researchers and increase the number, impact, and quality of international publications authored by Turkish researchers. Considerable percentages of papers with Turkish affiliations were supported in the early years of this program, even though the rate of support has gradually decreased (to
The total amount of support was about 124 million Turkish Liras (in 2015 current prices; equal to
The support program has been in place for almost a quarter century. Yet, its impact has not been evaluated in the past. We have been asked by TÜBİTAK to evaluate the effectiveness of the program and given the payment records of 157,000 supported publications. They included, among others, journal information (name, year, its class based on Journal Citation Reports’ subject categories), type of contribution (e.g., article, review) and the amount of support.
Based on the payment records provided, the characteristics (i.e., impact factors) of journals in which supported papers with Turkish affiliations appeared have been analyzed, the functioning of the support algorithm has been studied, and the effectiveness of the overall support program has been evaluated. Findings indicate that the authors of mediocre papers published in journals with relatively low impact factors have mostly been supported due to the use of skewed distributions of journal impact factors in determining the amount of support. The existing support algorithm, on the other hand, does not seem to function as conceived.
This paper presents only the findings of the interrupted time series analysis with a view to find out if the support program has had any impact on the increase of the number of papers with Turkish affiliations. It is organized as follows: The Literature Review section briefly discusses the findings of relevant studies including those that provide some background on the Turkish case. The Data Sources and Method section describes the data used and provides information on interrupted time series analysis. The detailed findings are presented thereafter (Findings and Discussion) along with the limitations of the study. The paper ends with Conclusions.
Performance-based research funding systems (PRFSs) came into being in the 1980s. Based on rewarding the outputs, the rationale of PRFSs is to provide more support to institutions (or individuals) with higher performances so that the ones with lower performances will strive to improve theirs in order to get more support (Herbst, 2007). Yet, it is not clear if PRFSs based on outputs and competition increase the scientific productivity and the impact of outputs. In a relatively recent study comparing PRFSs and outputs of eight countries, countries with less competitive PRFSs such as Denmark turned out to be as effective as the ones with more competitive PRFSs such as the UK and Australia (Auranen & Nieminen, 2010). Some researchers drew attention to the potential “side effects” of PRFSs based on competition, as they tend to “homogenize” research outputs, discourage experiments using new approaches, and reward researchers playing “safe” even though their contributions may not have any societal impact (Geuna & Martin, 2003). The idea of increasing productivity on the basis of outputs and competition seems more complicated than what decision-makers have initially thought (Auranen & Nieminen). For instance, there appears to be some evidence (albeit with relatively small effect sizes) that China’s “cash-for-publication” policy tends to increase researchers’ productivity (Heywood, Wei, & Ye, 2011). Yet, such cash incentives for publications that are in effect in China, South Korea, and Turkey seem to increase the number of submissions but are negatively correlated with acceptance rates (Franzoni, Scellator, & Stephan, 2011). Based on the countries of the first authors of papers submitted to the journal
There are mainly two types of PRFSs in use: (1) the ones based on peer review or informed peer review supported with bibliometric measures; and (2) the ones based solely on bibliometric measures such as journal impact factors. The UK’s Research Excellence Framework (REF) is the largest research assessment system in the world (De Boer et al., 2015). Based on peer review, REF has been used since 1986 to distribute funds to research institutes and universities on the basis of their performances. Despite their shortcomings, PRFSs based on bibliometric measures only are on the rise, as they are, in comparison to peer review, easier and less costly to apply as a “proxy” to assess performance. Therefore, they tend to get preferred by increasingly more countries lately.
PRFSs and publication support systems based on bibliometric measures generally use the number of papers published in refereed journals and their impact in terms of citations as the main criteria to determine the research institutes and researchers to be supported. Impact factors (IF) and article influence scores (AIS) of journals are the two most commonly used metrics.
Journal IF was originally proposed by the late Eugene Garfield (1972) to help librarians in their selection of journals for subscription. It is an indicator of the quality of a journal in general and measures the citation impact of an “average” paper published therein. It does not say anything about the quality of an individual paper in that journal and how many citations, if any, it would gather in a certain period of time after its publication (e.g., two years).
Citation distributions used to calculate the IFs of journals are quite skewed, indicating that few papers published in a given journal get cited much more frequently while the majority get unnoticed or rarely cited (Marx & Bornmann, 2013). This is the case even for the most prestigious journals with the highest IFs such as
Number of publications with Turkish affiliations (1976–2015).Year Papers Other Total Year Papers Other Total N % N % N N % N % N 1976 216 80 53 20 269 1996 3,359 84 623 16 3,982 1977 229 72 91 28 320 1997 3,844 83 796 17 4,640 1978 272 72 108 28 380 1998 4,460 82 1,001 18 5,461 1979 256 71 106 29 362 1999 5,201 83 1,078 17 6,279 1980 343 74 123 26 466 2000 5,462 84 1,059 16 6,521 1981 299 73 110 27 409 2001 6,684 84 1,271 16 7,955 1982 315 70 132 30 447 2002 8,985 86 1,434 14 10,419 1983 354 72 141 28 495 2003 10,662 84 1,978 16 12,640 1984 420 77 129 23 549 2004 13,199 84 2,488 16 15,687 1985 447 76 145 24 592 2005 14,194 83 2,877 17 17,071 1986 506 77 151 23 657 2006 15,070 79 4,099 21 19,169 1987 588 77 174 23 762 2007 17,853 80 4,414 20 22,267 1988 672 75 227 25 899 2008 19,327 82 4,379 18 23,706 1989 829 80 209 20 1,038 2009 21,655 82 4,627 18 26,282 1990 912 78 261 22 1,173 2010 22,833 83 4,760 17 27,593 1991 1,134 80 290 20 1,424 2011 23,588 82 5,325 18 28,913 1992 1,351 77 406 23 1,757 2012 25,254 82 5,607 18 30,861 1993 1,519 76 482 24 2,001 2013 26,526 79 7,200 21 33,726 1994 1,754 73 643 27 2,397 2014 27,242 79 7,315 21 34,557 1995 2,233 72 885 28 3,118 2015 28,662 79 7,530 21 36,192
Number of papers supported by TÜBİTAK (1997–2015).Year # of papers supported by TÜBİTAK # of papers with Turkish affiliations (WoS) Percentage supported (%) 1997 2,247 3,844 58 1998 2,657 4,460 60 1999 3,088 5,201 59 2000 3,298 5,462 60 2001 4,216 6,684 63 2002 5,888 8,985 66 2003 7,517 10,662 71 2004 9,511 13,199 72 2005 7,036 14,194 50 2006 8,122 15,070 54 2007 10,551 17,853 59 2008 10,411 19,327 54 2009 11,554 21,655 53 2010 11,592 22,833 51 2011 9,574 23,588 41 2012 10,641 25,254 42 2013 10,203 26,526 38 2014 10,257 27,242 38 2015 8,014 28,662 28 Total 146,377 300,701 49
PRFSs are reviewed by several researchers (e.g., De Boer et al., 2015; European Commission, 2010; Geuna & Martin, 2003; Hicks, 2012). Most EU countries, Norway, USA, Australia, New Zealand, and China have some PRFSs in place. We provide a few examples of PRFSs that either solely use journal IF or use it in combination with peer review (excluding the ones based only on peer review such as REF in the UK).
Italy uses a PRFS where an expert panel decides whether to use citation analysis or peer review (or both) for each publication. Universities are ranked on the basis of a quality score consisting of citations and other journal metrics, which determine the amount of support each university gets. Some 30% of the research funds are distributed according to the outcome of this evaluation (Abramo, D’Angelo, & Di Costa, 2011; Abramo & D’Angelo, 2011, 2016).
Similarly, Spain uses a mixed system, although researchers are encouraged to publish in journals that are listed in the top quarters of JCR’s subject categories. Researchers who publish in such journals receive monetary support that ranges somewhere between 3% and 15% of their monthly salaries (Osuna, Cruz-Castro, & Sanz-Menéndez, 2011).
A number of countries such as Czech Republic, China, Finland, and Australia use journal IF exclusively to support research institutes and individual researchers. Finland, for instance, linked journal IF directly with research support by legislation (Adam, 2002). Similarly, Australia and the Czech Republic make direct linkage between research evaluation and funding by counting scholarly outputs and assigning a score to each on the basis of bibliometric measures. These scores are then used to determine the amount of monetary support and papers that appear in refereed journals or journals with relatively higher IFs get the highest scores (Butler, 2004; Butler, 2003; Good et al., 2015, Table 3). Norway also has a similar system based on weighting journals on the basis of various criteria and created three different journal lists (Schneider, 2009). China, on the other hand, uses journal IF most comprehensively in that academic recruitments and promotions, university rankings (and the amount of research support they get), support of Chinese journals that are listed in Chinese Citation Indexes all rely on journal IFs. The procedure seems to have been automated, as a researcher publishing in a journal with a certain IF knows how much support s/he would get. For instance, the author of a paper published in a journal with IF higher than 15 receives 300,000 Yuan (
Test statistic (Ljung Box).Model Statistics Model Number of Predictors Model Fit statistics Ljung-Box Q(18) Number of Outliers Stationary R-squared Statistics DF Sig. Makale sayısı-Model_1 3 .607 23.531 17 .133 0
Turkey is no exception: journal IFs are considered as an indicator of quality and have been used as an important criterion in academic promotions since early 1990s. In addition to individual universities, TÜBİTAK has initiated a nationwide monetary support system based exclusively on journal IFs. Journals classified under Q1, Q2, etc., in JCR’s subject categories have been used to determine the monetary compensation (Tonta, 2015). More recently (2016), Turkish Higher Education Council (HEC) started a new support scheme based mostly on journal IFs and the faculty whose scores are above a certain threshold in terms of number of academic activities (mostly publications) during the previous year get an additional 10% to 15% on top of their regular monthly salaries throughout the year (Akademik, 2015).
It should be noted that performance-based research funding and publication support systems based on quantitative measures tend to have some adverse effects. Researchers seem to adjust to the requirements very easily and change their publication patterns and behaviors. Such systems are prone to “gaming,” too, and researchers become more “opportunistic” (e.g., publication “inflation”) and less ethical (e.g., “fake” citations) in time. Unintended consequences of PRFSs in several countries (e.g., Australia, Czech Republic, and Spain) were reported in the literature (Butler, 2003; Butler, 2004; Good et al., 2015; Osuna, Cruz-Castro, & Sanz- Menéndez, 2011; Tonta, 2014). For example, more papers tend to get published in journals with relatively lower IFs. A similar trend has also been observed in Turkey (Kamalski et al., 2017;Önder et al., 2008; Yurtsever et al., 2001, 2002). As the Goodhart’s Law states, “When a measure becomes a target, it ceases to be a good . measure.”
It should also be noted that correlation between competitive PRFSs and the research productivity is not clear-cut (Auranen & Nieminen, 2010). Excessive competition seems to reduce the time and energy otherwise to be expended for research. In this paper, we test the conjecture if TÜBİTAK’s publication support system has had an impact on the increase of number of publications listed in citation indexes with Turkish affiliations.
We performed a search on Web of Science (WoS) (December 19, 2016) to identify all the publications with Turkish affiliations listed in Science Citation Index (SCI), Social Sciences Citation Index (SSCI) and Arts & Humanities Citation Index (A&HCI) between 1976 and 2015. More than 390,000 records were retrieved, 81% of which were full papers (articles) while the rest were other types of publications (e.g., reviews, notes, and letters to the editor).
TÜBİTAK provided the payment data for about 157,000 supported publications (93% of which were papers). These records were first cleaned, then coded as either “full papers” (articles) or “other” types of publications, classified under various criteria (e.g., year, class of journal, amount of support paid), ranked and combined, if necessary.
We used MS Excel and SPSS 23 for the detailed analysis of data and prepared both WoS and TÜBİTAK records for interrupted time series analysis outlined below (Interrupted, 2013). (See Appendix A for time series data prepared for interrupted time series analysis.)
The interrupted time series (ITS) analysis technique (also known as quasiexperimental time series analysis or intervention analysis) is used in this paper to measure the impact of TÜBİTAK’s support program. ITS analysis measures if an “event” occurring at any given stage has an immediate or delayed effect on the time series data. For instance, an unexpected political development in a given country may increase the exchange rates, or a terrorist attack may reduce the number of tourists. These “events” (called “interventions”) may be planned or not planned. As ITS analysis is a quasi-experimental method, it is possible (by means of using a control group) to verify if the change has occurred because of the intervention.
ITS analysis is based on the following statistical model:
where
states that there is no statistically significant difference between the levels of series before and after the intervention (i.e., it has no impact on dependent variable (McDowall et al., 1980). It is assumed that the parameters in time series models stay the same before and after the intervention and that no other events that affect the parameters take place. ITS analysis can be applied to both static and dynamic (“ergodic”) time series. The ARIMA model is used for non-static series whose arithmetic means, variances, and co-variances change as time passes. This model is expressed as ARIMA (
We have WoS data of publications with Turkish affiliations (1976–2015) and data of supported publications by TÜBİTAK (1997–2015). The program (“intervention”) started in 1993 and enough data points exist both before (1976–1992) and after (1993–2015) the intervention so as to be able to apply ITS analysis to time series data (Cochrane, 2002).
It is not always easy to determine when the performance-based funding system in a given institution is exactly introduced and how long it takes for the system to start to have some impact on the publication output of that institution (van den Besselaar, Heyman, & Sandstrom, 2017; Butler, 2017; Hicks, 2017). We took the date of the decision of TÜBİTAK’s Scientific Board to initiate the support program (June 12, 1993) as the starting date. As relatively fewer researchers benefited from the support program in the early years, we thought that the effect of the program might be observed with some delay (lag) and therefore measured its delayed effect one (1994), four (1997) and 10 years (2003) after of its start.
We have no data on papers (full articles) whose authors have not been supported. However, a relatively small group of authors of other types of contributions can function as a control group, as only 3% of the total amount of support on average was set aside for such contributions even though 19% of publications were of such nature. The authors of other types of contributions were paid half of what the authors of the full papers were, and a mere 1% of the support budget was allocated to them in 2013, for example. This percentage should ideally be 0 (zero) in order for it to function as a true control group. Yet, we think that it can be used as a control group with some caution and the generalization should be interpreted accordingly.
The descriptive data about the number of papers and the total number of publications originating from Turkey are presented in Table 1 and Figure 1. The rate of increase is quite steep, especially starting from 2000s. This rate of increase made Turkey in those years one of the fastest growing countries in the world in terms of number of papers, and Turkey moved up the ladder very quickly from 45th in 1983 to 25th in 1999 to 18th in 2008 in the world, contributing to 1.56% of the overall scientific production in the world.
A considerable percentage of these publications were supported by TÜBİTAK’s support program when it was first initiated in 1993. However, the support program seems to have not kept up with the pace of increase of papers and the percentage of papers supported went down from 70% in early 2000s to below 30% in recent years (Table 2, Figure 2).
The detailed analysis of changes in TÜBİTAK’s support policies over the years is beyond the confines of this paper (see Tonta, 2017b). Instead, we concentrate on whether TÜBİTAK’s support program has actually played a role in the steep rate of increase of papers by Turkish researchers. The time path of the number of papers listed in the Web of Science (WoS) originating from Turkey between 1976 and 2015 is given below (Figure 3). The intervention point (1993) is marked on the graph. As there exists a trend of increase in the number of papers both before and after the intervention, we took the difference of the time series from the 1st level (
We then defined ARIMA (1, 1, 0) model for interrupted time series data and wanted to see the impact of TÜBİTAK’s support program in 1994, 1997, and 2003 (after one, four and 10 years of its start, respectively). The test statistic of the ARIMA model shows that the defined model is suitable for the time series data (
ARIMA Model Parameters.Estimate SE t Sig. # of papers # of papers No transformation Constant AR Lag 1 -57.138 334.811 -.171 .866 Model 1 .153 .170 .899 .375 Difference 1 Time series No transformation Numerator Lag 0 14.051 29.910 .470 .642 Before/after No transformation Numerator Lag 0 11.258 708.202 .016 .987 Support Program Effect No transformation Numerator Lag 0 29.091 36.715 .792 .434
In order to see the effect of the support program on the number of papers with Turkish affiliations, we continued with this model. The slopes of pre- and postintervention are the same for all analyses. It is possible to see the direct effect of the intervention on the number of papers with Turkish affiliations (Table 5). According to the model, an additional 564 papers were published in 1994 because of the support program. However, the effect of the support program is not statistically significant (
Values showing the delayed effect of TÜBİTAK’s support program.Year Predicted increase SE 1994 563.633 390.084 1.446 .157 1997 651.241 431.129 1.510 .140 2003 825.784 571.279 1.446 .157 2015 1,174.941 947.761 1.240 .224
Despite the fact that other types of contributions (non-papers) have been supported very little during the period of analysis, the rate of their increase is on a par with that of generously supported papers (see Figure 5). Slopes of linear regression lines of papers and non-papers are almost identical with corresponding
It should be noted though that interrupted time series analysis has some limitations. The assumption that no other “event” or “events” occurred during the period of analysis that might have affected the time series data is one of them. For example, the prerequisite of having papers published in journals listed in citation indexes for academic promotion may have triggered this increase, as more than 90% of research in Turkey has been carried out in universities, and the number of academic personnel in universities has increased tremendously over the years. Moreover, in addition to the number of research personnel in universities, the number of papers may be increasing due to a number of other factors such as the number of researchers per 10,000 capita, and the share of R&D expenditures within the Gross National Product (GDP). As indicated earlier, even though some positive correlation between PRFSs and the number of papers has been observed, this may not necessarily point to a strong causality between the two. As was the case in Spain (Osuna, Cruz-Castro, & Sanz-Menéndez, 2011), the number of papers with Turkish affiliations continues to increase perhaps not because of TÜBİTAK’s support program but because of other factors such as the growth in and the maturity of universities’ research systems including academic personnel.
We should also note that the interrupted time series analysis tells us whether the intervention has had any significant effect on the dependent variable or not but it does not tell us what caused the increase in the number of papers if it was not the intervention. To find out this, we carried out a multiple regression analysis and observed fairly strong correlation between the number of papers with Turkish affiliations and the number of academic personnel as well as the number of supported papers. However, we decided not to report the results of the multiple regression analysis, as the Durbin-Watson statistic was rather small (0.921), probably indicating the existence of serial autocorrelation between variables and thereby making the results less reliable. This can to some extent be observed from Figure 2: the correlation between the number of papers with Turkish affiliations and the number of supported papers was positive and statistically significant between 1997 and 2006 whereas it was negative and not statistically significant between 2007 and 2015.
For a more definitive answer to the question of whether TÜBİTAK’s support program has had any effect on the increase in the number of papers with Turkish affiliations, a true control group is needed. In other words, the rate of increase of papers supported by TÜBİTAK needs to be compared with that of non-supported ones. Even such a comparison may not necessarily be sufficient to reveal the causality, should there be any, between TÜBİTAK’s support program and the steep increase in the number of papers with Turkish affiliations. For this, individual level data for both TÜBİTAK-supported and non-supported papers are needed to see if the increase is due to the increase of the productivity of: (1) the same researchers benefiting from TÜBİTAK’s support program; (2) more researchers responding to TÜBİTAK’s cash incentives; (3) researchers who have not sought TÜBİTAK support for their papers in the past at all; or (4) a combination of some or all of the above.
As part of TÜBİTAK’s support program, the authors of over 157,000 publications received more than 124 million Turkish Liras (in 2015 current prices,
We should point out that the objective of the support program is not to increase the number of papers
Such support programs should function as a leverage to speed up the scientific and economic development of countries.A thorough study as to why the support program did not seem to function as intended should be carried out. After a comprehensive review of existing support programs, new policies should be instituted to increase the impact and quality of scientific papers originating from Turkey, and TÜBİTAK’s support program should be redesigned accordingly.
Based on 25 years’ worth of payments data, this is perhaps one of the first large-scale studies showing that “cash-for-publication” policies or “piece rates” paid to researchers tend to have little or no effect on the increase of researchers’ productivity. The main finding of this paper has some implications for countries wherein publication subsidies are used as an incentive to increase the number and quality of papers published in international journals. They should be prepared to consider reviewing their existing support programs (based usually on bibliometric measures such as journal impact factors) and revising their reward policies.