Substantiality: A Construct Indicating Research Excellence to Measure University Research Performance

In this paper, the following procedure shows that substantiality can play an important role as a construct for measuring the research performance of research universities. First, through an analysis of the recent improvements in the reputation rankings of Tsinghua and Beijing Universities, it is shown that there is an aspect of research performance of research universities that cannot be well captured by the conventional indicators of quantity and quality. Second, we clarify the characteristics of substantiality proposed in this paper using thought experiments and real data; then we confirm that it can measure some characteristics of research performance that are difficult to capture by indicators of quantity and quality. Afterward, it is confirmed that the reputation mentioned above ranking problem can be explained at least consistently by substantiality indicators. Finally, we demonstrate the “predictive power” of substantiality indicators on the research reputation of universities.

The subtitle of the announcement of the Times Higher Education (THE) World University Rankings 2019 was to convey China's leap forward: “China is now home to the best university in Asia, while France's Sorbonne University is the highest-ranked newcomer in the table.” The paragraph at the beginning of the text also conveyed the success of Chinese universities, especially that of Tsinghua University. The development of Chinese universities is not a projection anymore but a reality that has arrived.

In 2016, China produced the largest number of academic publications compared to any country

National Science Foundation, “Science and engineering Indexes 2018,” [Online]. Available: https://www.nsf.gov/statistics/2018/nsb20181/.

For example, in the reputation rankings based on the results of questionnaire surveys targeting selected academics and researchers published by Times Higher Education, the reputation ranks of Tsinghua and Peking University's research were 37^th and 49^th, respectively, in 2011

https://www.timeshighereducation.com/world-university-rankings/2011/reputation-ranking#!/page/0/length/50/sort_by/scores_research/sort_order/asc/cols/undefined (retrieved on 28/06/2021).

As previously mentioned, there are two types of commonly used research performance indicators. One type measures quantity, such as the number of publications, the other type measures quality based on citations, such as the ratio of highly cited publications (e.g. top 1% most cited publications) and field-weighted citation impact (FWCI)

Elsevier (2014) [Online]. “SciVal metrics guidebook,” Available: http://www.elsevier.com/__data/assets/pdf_file/0006/184749/scival-metrics-guidebook-v1_01-february2014.pdf (dead link; retrieved on 27/12/2020).

Considering that surveys of reputation rankings are conducted in the first quarter of each year, we studied the top 50 universities in terms of research reputation by using the previous year's quantity and quality indicators in a five-year publication window. Among the 36 universities ranked higher than Tsinghua University in 2011, 26 universities published less. Meanwhile, 35 of 36 universities had higher FWCIs than Tsinghua University. However, no university had a lower FWCI than Tsinghua University among universities ranked lower than it. In 2017, 12 out of the 13 universities that ranked higher than Tsinghua University had fewer publications than Tsinghua University, and only one university out of the 13 universities had a lower FWCI. Meanwhile, although 34 out of the 36 universities ranked lower than Tsinghua University and published less, only four universities had a poorer FWCI than Tsinghua University among 36 universities. Peking University is also in a similar situation.

Accordingly, neither indicator can explain the significant improvement in the rankings of the two universities. Although it is certain that the research performance of each university has greatly improved as far as the two indicators show during this period, it is hard to say that the quantity and quality of research outputs are decisive for its recognition as a top university. Rather, it makes the current problem easier to understand if we focus on the concept of substantiality, which we propose as the third construct indicating research excellence to evaluate university research performances.

We have demonstrated that there is a feature of research universities that can hardly be grasped by the traditional quantity and quality indicators alone.

Now let us compare the research performance in scientific field X between University A and University B, as shown in Figure 1 A. Each blue circle represents a single publication, and its number is its citation count. One publication from University A was cited 52 times, but the other publications were not very much cited. However, University B has fewer publications and no highly cited publications, but its publications are well-cited overall. Which university has better research performance in the scientific field X? There are some paths of possible opinions. There are reasons to think that University A, which has a highly cited publication, has better research performance, while University B could be perceived as better for having a good average of citations despite not having a highly cited publication.

Figure 1

However, the quantity and quality indicators, such as the number of publications, the total number of citations, and an average number of citations like FWCI, show that University A is superior to University B. From these conventional metrics, University B's consistent research capabilities are not evident.

Therefore, we believe that there is a need for a new way to evaluate rich and profound research capabilities, such as those of University B. We define this research capability as substantiality, accumulation of excellence (i.e. excellent players) to produce a certain volume of publications at a certain level of quality.

The Japanese word ATSUMI

ATSUMI is a term used in IGO game (a famous East Asian board game) and is an important concept to decide strategy and tactics in the game. Basically, the term means influence and strength of an arrangement of game pieces, that is, stones (Yokogawa, Nishino & Mizuno 1995). Some people say that the stones represent soldiers (i.e., players) in the game world.

For example, the ARWU ranking (the Shanghai Jiao Tong University ranking) uses the number of highly cited researchers, which we can consider to be a substantiality indicator because it indicates the quantity of human resources with certain abilities in each university. Also, we consider that the number of highly cited publications (e.g. number of top 1% most cited publications) can be regarded to represent substantiality as estimated by the outcome.

In addition, as another substantiality indicator, we propose h-index for evaluating institutional group performance in the past five years (so that only recent history is included in the substantiality concept). We call this the institutional h5-index. Applying it to University A and University B (Figures 1 A and 1 B), in listing the publications with citations, the h5-index is three for University A and six for University B. Using this index, the stable and consistent characteristics of the research performance of University B can be clearly captured.

Applying the institutional h5-index to actual university data, Figure 1 C lists publications from Kyushu University and Shinshu University in Japan in descending order of the number of citations up to a total of 150 publications. Kyushu and Shinshu Universities have the same number of highly cited publications, but Kyushu University has publications in the second tier with more citations. In contrast, Shinshu University showed a higher FWCI. Although FWCI is an excellent indicator for evaluating the average quality, it is disadvantageous for Kyushu University, which publishes a larger number of publications. Using the institutional h5-index as well, it becomes clear that Kyushu University has more substantiality than Shinshu University. As an independent third axis, the h5-index does not necessarily correlate with FWCI. In fact, their correlations differ according to the field of science (Table 1).

Furthermore, we calculated the rank correlation coefficients among the indicators for research universities in Japan and the world (Table 2). Although the three substantiality indicators are strongly correlated with the number of publications, their correlations with quality indicators are quite different from those of the number of publications. Accordingly, substantiality indicators are intermediate in nature between quantity and quality indicators. Thus, a substantiality indicator such as the institutional h5-index evaluates an aspect of research performance that is different from those evaluated by the conventional indicators.

Regarding reputation rankings, substantiality can explain the leaps of the two aforementioned Chinese universities better than other indicators. The correlations of substantiality indicators (i.e. h5-index and number of top 1% most cited publications (Kutlača, 2015)) to research reputation scores in the top 50 universities are clearly higher than those of the number and FWCI of publications (Figure 1 D). Between 2011 and 2017, in terms of the number of top 1% most cited publications, Tsinghua University rose from the 40^th to 20^th and Peking University from the 39^th to 17^th among the top 50 universities in terms of research reputation. We can interpret these results as showing that the presence of the two universities in academic societies has become more noticeable due to the increase in substantiality. We pursue this finding further in the next section.

We have introduced the concept of substantiality and proposed indicators to evaluate the accumulation of excellence of research capabilities and organizations. We showed that there is a feature of research organizations that is difficult to grasp with previous indicators or combinations thereof, and we have clarified that substantiality indicators capture the feature reasonably well. Even if substantiality indicators capture such a feature well, however, the question of what kinds of benefits it brings needs to be answered.

We demonstrate the “predictive power” of substantiality indicators on the research reputation of organizations. We focused on the relations between substantiality indicators and research reputation because we had the following hypothesis (Figure 2)

As a matter of fact, it is impossible to verify this hypothesis because world reputation rankings have been announced in just the past nine years, and their scores and detailed ranks have been disclosed for only the top 50 universities.

Figure 2

Reputations were used as evaluation items in two famous world university rankings (i.e. THE and QS rankings), and their weights were virtually the largest among their evaluation items. Given the situation where research evaluation by peer reviews is still important (e.g. Hicks, 2015), the emphasis is placed on research reputation in the university rankings, which can be regarded as peer reviews of a large-scale version.

As it is the research reputation that summarizes subjective evaluation results obtained through a comparatively large-scale survey, it is unlikely that people, independent of the ranking agencies, understand what it means. Furthermore, it is not clear how to raise the research reputation.

Then, we analyze the relations between the research reputation in Times Higher Education's world reputation rankings (2011–2018)

https://www.timeshighereducation.com/world-university-rankings/2018/reputation-ranking (retrieved on 28/12/2020).

We calculated Spearman's rank correlation coefficients

The distribution of scores of research reputation is so skewed that we have to use rank correlation coefficients.

Figure 3

Figure 4

As shown in these figures, the results are consistent with our hypotheses. Substantiality indicators are the most strongly correlated to research reputation, and older indicators tend to be more strongly correlated with research reputation. The time lag may be longer than expected. Giving that daily reputation in everyday life often depends on past actions, this result is not surprising. In any case, as we cannot secure enough trials, we cannot discuss the hypotheses too far. Still, their relations with research reputation could be suggestive for using substantiality indicators in practice.