Does a Country/Region’s Economic Status Affect Its Universities’ Presence in International Rankings?

Universities, as centres for higher education and research, have long vied quietly and nearly invisibly for the top positions on the international arena. The grounds on which universities compete to attract students and raise funds for their scientific activities include reputation, prizes awarded to professors and students, and contributions to significant discoveries. The first world ranking of universities was formulated by what was then Shanghai Jiao Tong University’s Institute of Higher Education (now the Graduate School of Education) (Cheng & Liu, 2007; Liu & Cheng, 2005; Liu, Cheng, & Liu, 2005). It was soon followed by others such as the Times Higher Education Supplement’s QS World University Ranking (Times QS ranking) (Buela-Casal et al., 2007) and the Leiden ranking in 2007 (Waltman et al., 2012), which measures the results of world universities’ scientific and teaching performance. These rankings gave rise to another much more complex scenario in which universities shifted from local competition, i.e. among institutions in the same country/region, to compete globally, with comparisons crossing national borders (Azman & Kutty, 2016; Kauppi, 2018; Marginson, 2007; Musselin, 2018; Ordorika & Lloyd, 2015). Higher education institutions were therefore driven to update their objectives to adapt to the new situation, which included improving their positions on rankings in a bid for prestige as research bodies (Lim & Øerber, 2017; Millot, 2015; Zhang, Bao, & Sun, 2016). Such competition was heightened by governments’ need to enhance access to higher education and strengthen their countries/regions’ presence on the list of the most highly reputed universities (Bornman, Mutz, & Daniel, 2013; Guironnet & Peypoch, 2018; Musselin, 2018). Some governments have implemented different types of initiatives to support the internationalisation of their higher education institutions.

Each ranking is generated in keeping with a specific methodology, using bibliometric and other types of indicators. A given institution’s position may vary depending on the methodology used. In the Shanghai Jiao Tong University (ARWU) and Leiden rankings, classification prioritises research, whereas the Times Higher Education (THE) list stresses reputation, measured on the grounds of questionnaires sent out to reputed academics in different areas (Ordorika & Lloyd, 2015; Shin & Toutkoushian, 2011).

According to Safón (2013), the most influential global rankings are ARWU, THE and QS, whose methodological differences have prompted studies aiming to identify the strengths and weaknesses of each. ARWU, based primarily on research and academic performance, consists of six indicators, including the award of prestigious distinctions such as the Nobel Prize to students or faculty. Safón (2013) contends that this indicator favours older institutions in countries/regions with long ranking traditions. THE in turn, uses now 13 performance indicators one of them related to academic prestige, assessed on the grounds of large-scale surveys. It had been criticised for its regional bias, now it take care to get a better sample of academics from over the world.

Several authors have analysed and compared the methodologies used by ranking institutions (Buela-Casal et al., 2007; Van Raan, 2005). Marginson (2007) compared the scientific activity of Australian universities listed on the ARWU and THE rankings, identifying bias in the methods used by both and hence in their results. He deemed that combining subjective reputational data with objective research data, as in the THE, was not a valid approach, and also detected a strong tendency in the ARWU methodology to favour universities in English-speaking countries/regions. Aguillo et al. (2010) compared five well-known rankings, ARWU, THE, Leiden, WR (Web Ranking of World Universities (for 2005–2008)

http://www.webometrics.info/

In their statistical analysis of known rankings, Bornmann, Mutz, and Daniel (2013) explored one of the main bibliometric indicators used by the Leiden ranking, namely the proportion of papers published by a university that lies within the 10% most cited (PP_top10%). These same authors pointed out that a more sophisticated statistical model than deployed by the editors of the Leiden ranking

http://leidenranking.com/leidenranking.zip

Taking the variable PP_top10% as a basis, Bornmann, Mutz, and Daniel (2013) analysed several significant questions in connection with the Leiden ranking, the first being the accuracy of the positions and stability intervals predicted by the multi-level regression model proposed. Another question posed related to whether differences in the impact factor explained scientific research rankings and whether a relationship could be found between such differences and the country/region where the university is located (Bornmann & Moya-Anegón, 2011). A final issue addressed by these authors was whether inter-university differences could be explained by economic factors (Per Capita Gross Domestic Product (GDP PC), Gross Domestic Product (GDP), a country/region’s total area and population) or by the number of papers published by a given university.

Marginson (2007) used statistical methodology to compare each country/region’s economic status, calculated as GDP and GDP per capita, to the number of its universities in the ARWU top 100 and top 500. In another paper, Marginson and van der Wende (2007) compared each nation’s share of world economic capacity against the proportion of research universities in the ARWU top 100 and top 500.

Docampo (2008) analysed the impact of the ARWU since its first edition and reviewed the criticism and updates of its methodology for listing universities. The author also aggregated the data for each country/region’s universities and weighted the size of national economies measured as the share of each one’s gross domestic product (GDP) in the world total in 2006.

Other authors, including Kempkes and Pohl (2010) and Johnes and Yu (2008) analysed German and Chinese universities’ activity on the grounds of R&D input and output, using data envelopment analysis (DEA) to determine their efficiency and total productivity. Rhaiem (2017), makes a systematic review about research efficiency. Using several electronic databases, the author brings a set of inputs and outputs related to this issue. In the same way, Barra et al. (2018), measures the efficiency of Italian higher education using parametric (Stochastic Frontier Approach) and non-parametric (DEA) methodology.

The primary aim of the present study is to analyse the relationship between a number of socio-economic indicators for selected countries/regions and performance measured as the number of their universities positioned in the ARWU top 100 and top 500. Multiple regression techniques were used for this analysis. A second objective was to identify how the indicators analysed would have to change for a country/region to raise the number of its universities listed and their position on the ARWU. The reasons for choosing ARWU rankings were similar to the explanations provided by Marginson (2007), Docampo (2011, 2012, 2013), and Ordorika and Lloyd (2015), i.e. because they “are credible, based on solid, transparent numerical data…”, and “because it is the only global ranking that focuses on research activities of the universities…”. The properties, strengths, weaknesses and reliability of the ARWU scale were also studied by Docampo (2008, 2011). This author contended that the emphasis on international publications and citations (per year and cumulative) may bias the ranking, favouring the English language and traditional institutions. Docampo also noted, however, that aggregating data for the sciences by country/region smooths the effects of these indicators. The use of indicators that take the distinctions awarded to university teachers and students into consideration has been addressed by a number of authors, including Docampo (2008) and Bornmann, Mutz, and Daniel (2013). Their argument is that history may weigh significantly on the results, enabling institutions to rank in the upper echelon on the grounds of just a few shining moments.

The present study differs from the ones referenced above in the multivariate approach adopted, which proposes multiple linear regression models and transformations of the predictor and response variables.

This section contains a detailed description of the data retrieved and the methods used for the analysis and the model proposed.

Data were aggregated for the 45 countries/regions with at least one university on the list of the ARWU 2012 top 500. Nonetheless, Table 1 gives only the 39 countries/regions that met the first condition (the total number of universities for the countries/regions selected was 492). The top 100 universities were located in 16 countries/regions, in which the United States clearly prevailed, with 53 of the top 100 and 150 of the top 500 universities, followed by the United Kingdom with 9 in the top 100 and 38 in the top 500.

The search of statistical relationships between related variables via linear regression analysis may be distorted by the presence of spurious correlations between regressors and the predicted variable. This effect may be more dramatic for analysis of cross-sectional nature as in the present case. To discard these suspects in our analysis we have searched correlations and partial correlations. As Table 2 shows, the correlation values obtained at the 10% significance level guaranteed the relevance of the regression analysis performed; consequently, a linear statistical relationship could be concluded to exist between the predicted variable and the proposed regressors.

3.1

Top 500 universities

After varimax rotation, the first two components with eigenvalues over 1, which accounted for 97.8% of the total variance in the original variables, were chosen for the data considered as a whole. The indicators for the general sample characteristics were grouped in the first component and all the others in the second. Table 4 gives the percentage of the variance for each indicator explained by the two components selected.

Table 3

Values for each indicator by component (before excluding outliers).

Principal Component Analysis
Indicator	Population	GDP	GDP PC	HDI
First component	-0.499	0.041	0.979	0.964
Second component	0.864	0.998	-0.099	-0.153
% of explained variance	0.996	0.998	0.967	0.952

Table 4

Values for each indicator by component (after excluding outliers).

Principal Component Analysis
Indicator	Population	GDP	GDP PC	HDI
First component	-0.377	0.119	0.970	0.946
Second component	-0.922	-0.992	0.011	0.153
% of explained variance	0.993	0.999	0.942	0.918

Excluding data for USA, ZAF, NOR, IND, and CHN.

The MLR model for the data taken as a whole yielded an adjusted determination coefficient (R²) of 73.9 %, with the ANOVA showing the model to be significant (the y-intercept was 1.72 and the component coefficients were respectively 0.037 and 0.931). The Durbin-Watson test (which measures inter-residual correlation) yielded a value of 2.077, indicating that the null hypothesis (ρ=0) could not be rejected and one of the scores for one of the principal component regressors was not significant. A robust M-estimator regression model was built for this sample using the same scores, a y-intercept of 1.78 and component coefficients of 0.042 and 0.940. Cook’s distance revealed the existence of outliers, which are shown in Figure 1.

Figure 1

The MLR model findings denoted the presence of high residuals due to the effect of possible outliers. Highly influential observations and residuals for USA, ZAF, NOR, IND, and CHN were detected and excluded and the principal components were recalculated and the first two components selected. The percentage of variance explained by the components in these new results amounted to 96.3% (Table 3).

The MLR model-adjusted R² rose from 0.739 to 0.787, the p-value was highly significant (p<0.001) and the Durbin-Watson value was 1.945. In this model, the dependent variable was the natural logarithm of the aggregate indicator “number of a country/region’s ranked universities”. The y-intercept for the model estimated under these conditions was 1.692 and the regressor coefficients were 0.111 and 0.781 for the first and second component scores, respectively. Both the ANOVA and the coefficients were significant at 90%.

The exponential functions of the dependent variable were calculated to compare the values estimated by MLR to the actual number of each country/region’s ranked universities. The result is shown in Figure 2. The regression findings can be graphically interpreted in terms of country/region positions with respect to the first quadrant diagonal. Countries/regions located above the diagonal, such as Japan, Russia, and Singapore, have fewer universities than predicted by the model on the grounds of their economic and social potential. The opposite is true for countries/regions such as United Kingdom, Germany, Spain, Italy, and France, which are located below the diagonal, for their real number of universities is higher than predicted.

Figure 2

Partial regression analyses were also constructed to ascertain the individual effects of each component on the dependent variable. The second component was found to have a positive effect, with an adjusted determination coefficient of 76.9% and a Durbin-Watson value of 2.03.

A robust M-estimator regression model was built using the same scores, with a y-intercept of 1.665 and component coefficients of -0.111 and 0.773. As in the preceding case, the number of each country/region’s listed universities was estimated by applying the exponential function to the dependent variable. Figure 3 plots the original values versus the values estimated with the robust MLR method. The graphs may be interpreted as in Figure 2: note that the country/region positions are largely the same in the two sets of figures.

Figure 3

When the HDI for each country/region in the ARWU top 500 was raised by 1%, the number of ranked universities did not rise in the same proportion. For some countries/regions, this change had no effect on their position on the diagonal. Examples are Mexico, Brazil, Hungary, and Poland. For others, in contrast, such as Finland, Spain, Germany, and United Kingdom, the 1% rise in HDI raised their values slightly, placing them above the diagonal (Figure 4).

Figure 4

Figure 5 shows how the model estimates varied when the GDP PC values were raised by 10%. Here, countries/regions such as United Kingdom, Italy, Canada, and Germany, and to a lesser extent Israel, Spain, Denmark, and Finland, were affected positively, with the rise in the expected number of universities in the ranking positioning them above the 45° line.

Figure 5

Figure 6 plots the estimated number of listed universities at the status quo versus the number when HDI and GDP per capita were both raised (by 1 and 10%, respectively). Here only Slovenia, Turkey, Hungary, and Czech Republic exhibited the same values in both cases, while the largest gains were found for countries/regions with high socio-economic indicators, such as United Kingdom, France, Germany, Canada, and Japan.

Figure 6

Lastly, the effect of variations in one of the scores on the model results was analysed. Figure 7 shows that a one-unit increase in the scores of the first component raised the model estimates by 5.7 units. For countries/regions such as Brazil, Mexico, Russia, and Spain the rises were 3, 3, 6, and 11 units, respectively. Japan and Germany, with outlying values of over 30 units, were positioned in the upper area of the figure.

Figure 7

This study analysed the socio-economic characteristics of countries/regions whose universities are listed on the ARWU from two approaches. The first consisted of representing these universities by means of a MLR model and a robust estimator, both based on principal components, while the second focused on the sub-set of countries/regions with institutions in the ARWU top 100.

Under the present assumptions, the fit to the present data set afforded by robust multiple linear regression was equivalent to the fit found with classical multiple linear regression. Only slight differences were observed in the MLR and robust MLR model results, as a comparison of Figures 2 and 3 shows. As in Safon (2013), countries/regions with long-ranking tradition such as Germany, United Kingdom, Canada, and Italy, had a larger number of universities in the top positions than predicted by the regression model. It could means that the competition among universities takes more a form of the “winners take the most” phenomenon. In contrast, for Japan, a country where social and economic performance is high, the number of ARWU universities projected by the model was much larger than the actual figure. These results were consistent with Docampo’s (2012) findings, “… suggesting that Japanese higher education system might have began to fall into decline in the past decade”. As in Docampo (2012), in this study the university systems in three Asia-Pacific countries/regions, namely Chinese Hong Kong, New Zealand, and Chinese Taiwan, were found to show good performance. Countries/regions such as Spain, Sweden, and The Netherlands exhibited higher real than predicted values, although as shown in Figures 2 and 3, they were close to the 45° regression line representing the expected behaviour further to the MLR model and a robust estimator.

Countries/regions with heavy investment in education, such as Japan and Denmark, had lower than expected results: i.e. in light of their social and economic status, they could intensify their presence on the ARWU.

Conclusions may also be drawn from the adjusted regression model estimates on how changes in the original variables might translate into improvements in positions on rankings. To that end, the effect of a change in the indicator on the response variable was studied assuming constant mean and variance, the statistics used to standardise the indicators in the principal component extraction phase.

When the HDI for each country/region in the ARWU top 500 was raised by 1%, the number of universities making the cut did not rise in the same proportion in all countries/regions. The values for some countries/regions, including developing nations such as Mexico, Brazil, and Turkey, as well as Hungary, Poland, and Portugal, remained unaltered. In other cases, the values rose only slightly, such as in Finland, Israel, Korea, and Spain. Growth in that number was proporionally higher in United Kingdom, Germany, Canada, Italy, France, Australia, and Japan.

When GDP per capita was raised by 10%, the number of universities estimated by the model to be included in the ranking failed to rise for certain countries/regions with per capita GDP below the median, such as Portugal, Argentina, Mexico, and Brazil. The number grew only slightly in some countries/regions with HDI and GDP per capita higher than the median, including Switzerland, New Zealand, and Finland. In contrast, according to the model findings, United Kingdom, Germany, France, and Australia would be positively impacted, i.e. the number of their universities expected to be on the ranking would climb.

When both welfare indicators were raised by 10%, most countries/regions exhibited significant rises in the presence of their universities on the ranking, especially countries/regions with the most favourable socio-economic conditions, such as United Kingdom, France, Germany, Canada, Australia, and Japan. Under these circumstances, the number of universities estimated by the MLR model remained unchanged only in Slovenia, Turkey, Hungary, and Argentina.

Similarly, the effect of altering the value of one of the principal component scores on the MLR model was also analysed: when that score for HDI and GDP per capita rose by one unit, the presence of each country/region’s universities on the ARWU rose by more than eight units.

To study the characteristics of the countries/regions with universities on the ARWU in greater depth, the focus was shifted to countries/regions with universities listed among the top 100, regarding this position to be an indicator of quality and excellence. To that end, a dummy variable was defined to separate the countries/regions analysed on the grounds of their presence or otherwise in this special sub-set. The analysis was inspired by the premise of Bornmann, Mutz, and Daniel (2013) to the effect that countries/regions with more abundant resources (GDP per capita) for implementing high quality projects may also have higher scientific research indicators, which are essential to positioning their universities at the top of the ARWU. Those authors also contend that the larger the number of universities engaging in research in a country/region, the more papers of excellence it produces. That, in turn, would favour their presence among the top 100.

According to Docampo (2008), the ARWU prioritises scientific research over reputation, where the dummy variable is also an indicator of excellence and hierarchy, inasmuch as having universities among the top 100 means that the country/region occupies a prominent position in research. A chi-square test was run to ascertain the relationship between this indicator and HDI and GDP per capita, duly classified as dummy variables. New indicators were defined in both cases: for HDI, whether a country/region’s index was regarded as high, and for GDP per capita, whether it was higher or lower than the median (an indication of the availability of economic resources). The former was identified as an indicator of social stability and the latter of the availability of financial resources. The results revealed a relationship for GDP per capita, but not for HDI. Further to those findings, a greater abundance of resources would favour the presence of universities in higher positions.

To verify the second premise put forward by Bornmann, Mutz, and Daniel (2013), a test was run to ascertain the existence or otherwise of a relationship between the number of universities per million inhabitants and the dummy variable, i.e. the presence or otherwise of each country/region’s universities on the ARWU top 100. Pearson’s chi-square test of independence between the two indicators yielded a value of p=0.000. That would confirm the existence of a relationship between research output and the number of universities per million inhabitants. Countries/regions that prioritize higher education exhibited the highest values for these indicators.