Methods and Practices for Institutional Benchmarking based on Research Impact and Competitiveness: A Case Study of ShanghaiTech University

ShanghaiTech is a research university established jointly by the Shanghai Municipal Government and the Chinese Academy of Sciences (CAS) in 2013. It is set up as a small-scale, internationalized, and first-class research institution aiming at tackling globally advanced and hard-fought scientific challenges. It focuses its research only on a few selected subject fields in Physical Sciences & Technologies, Biological Sciences and Technologies, and Information Sciences and Technology, and avoids trying to cover complete disciplines. As of June 2019, it has fewer than 200 faculty members, no more than 500 undergraduate and 1000 graduate enrollments yearly. The university evaluates the research performance of itself, its schools and institutes, and its researchers, not in numbers of papers or average citations, but on their research quality, impact, competitiveness, and power or potential of leading internationally. So the common ranking schemes do not serve its mission, and they are even unfair to it. Consequently ShanghaiTech has strived to design an individualized mission-oriented assessment scheme to see how it is fulfilling its mission, and what and how each subject areas are performing in this regard. The university also needs to be compared, in specifically chosen measurements, to those institutions it aims to surpass. It takes this effort seriously and invests in continuous efforts reporting directly to the president of ShanghaiTech University.

The following principles are established to guide the design of the scheme:

In this study, an evidence-based benchmarking effort is proposed and tested in practice; future improvements are proposed. These efforts are guided by the following assessment questions:

Many further questions can be explored to diagnose in-depth the how and why of reported strengths and weaknesses. However, it is felt that these explorations are better dealt with by research experts and decision-makers exploring the data interactively together with information analysts, including data outside the standard assessment databases.

Evidence-based assessment relies on objective indicators; these indicators must serve the principles and assessment questions, following a chain of analysis from the mission, principles, assessment questions, to indicators, and then further analytic results or “assessment answers”.

The key underlining indicators are those representing research quality and impact, focusing on the number of high quality papers and their citation impact. A set of indicators are selected to represent research quality and impact, such as the number of papers published in the Q1 list of SCI source journals, highly cited papers according to ESI, or papers published in Cell, Nature, and Science (CNS, a category often used in China). Research impact is measured using citation-based impact indicators, including the Category Normalized Citation Impact (CNCI) (Clarivate Analytics, 2018). Competitiveness is demonstrated by the percentages of high quality papers and high impact papers from each institution in the benchmarking line-up. Total numbers of publications are still used but only as a reference background. For some analyses, international and domestic medians are calculated and displayed for easy comparison with ShanghaiTech’s values.

It is fully recognized that publishing in high quality journals does not necessarily represent high quality of any single paper itself, but when averaged across institutions or disciplines, the average CNCI achieved by an institution may give an interesting picture of the overall quality of the institution. It is also recognized that for some fields, especially in computer science and IT, proceeding papers are important research outputs, and for other fields patents need to be included to present a full picture of research performance. Due to the focus on basic research of ShanghaiTech, and because this study aims to evaluate ShanghaiTech’s research in terms of basic research, the current effort employs only data related to research papers. Future iterations of the study intend to include data on proceeding papers and patents, though these present challenges as what to choose and how to evaluate their “impact”.

Table 1 gives the set of indicators following the framework in Figure 1. Nearly all of the analytic angles can be applied to different research units (schools or institutes, disciplines, or groups) to enable drilling down analysis, etc. Size of the entities may vary, but the majority of indicators employed are size-independent, enabling assessment of relative competitiveness.

Figure 1

As benchmarking is mission-oriented, the selection of benchmarking institutions is of the utmost importance. This selection must consider many factors of the targets, such as their nature, mission, scale, research emphases, etc. In line with the mission of ShanghaiTech to be a first-class research institution aiming at solving globally advanced and hard-fought scientific challenges, internationally top research universities must be chosen for benchmarking. Since ShanghaiTech is a Chinese university that will inevitably also be compared to top research universities in the country, a few top ones in China, which are internationally competitive, are selected too. Another consideration is the fact that ShanghaiTech is a brand new and experimental university of small-scale, internationalized, and selectively focusing on certain research areas. Hence, selected institutions must be as comparable as possible. Finally, to limit the workload of evaluators only a handful of institutions can be considered. A total of 8 international universities (Univ A—Univ H) are chosen from among the top 10 of the main world-wide rankings (e.g. U.S News, ARWU), and 5 domestic universities, selected from the top 5 among the Chinese institutions from the same rankings (Univ I—Univ M). Two newly established research-intensive universities are also added to the list (Univ N—Univ O). A research area-guided selection strategy, for those excellent in individual research areas similar to those of ShanghaiTech but not necessarily coming up high in ranking systems, was contemplated but is left to the second phase of this work due to workload consideration, though some on-request analyses of research competitiveness with selected niche institutions as benchmarking targets have been conducted for individual schools/institutes and even labs/teams.

Since research in ShanghaiTech is mostly in scientific fields and only high quality output will be included in the assessment, SCI-Expanded of WoS Core Collection was chosen as the data source, and further limited to article and review papers and ten research areas in the ESI schema, namely physics, chemistry, biology & biotechnology, materials science, engineering, computer science, molecular biology & genetics, immunology, microbiology, and neuroscience & behavior. The timespan covers the period from 2014 to the retrieval date in 2018 (Sep. 21^st) to coincide with the development period of ShanghaiTech. Some of the indicators can be obtained directly from data sources, while others are produced by InCites, and still others, such as the research content analysis to produce subject-area knowledge maps, is produced via analytic and visualization tools such as VOSviewer and Derwent Data Analyzer.

A large number of researchers from institutes of CAS work as adjunct professors in ShanghaiTech, and they publish papers with affiliations including ShanghaiTech. This makes it difficult to determine the contributions of different types of faculty members. For this reason three datasets of ShanghaiTech faculty were constructed. SHTech-A is the dataset for the entire faculty, full-time or adjunct, if any of the co-authors is from ShanghaiTech. SHTech-B is the dataset only for those authors in SHTech-A and which are first or corresponding authors. SHTech-C is the dataset consisting of those authors in SHTech-B which are full-time faculty members of the university.

First, indicators with high quality papers, high impact papers, and their percentages out of each institution’s total were calculated, presented in Table 2. Compared with the international leading universities Shanghai Tech had lower percentages of high quality papers, received fewer citations, and established less international collaboration. Yet, it was, by many of the indicators, better than most domestic universities. Interestingly, SHTech-C had higher percentages of highly cited papers and CNCI values, indicating that its full-time faculty performs quite well.

An analysis was conducted to see if the differences changed in various research areas. A ratio of papers in each discipline to the total publications of each institution was calculated. Knowing this ratio helps to put the numbers in perspective, avoiding comparing to those not really focusing on the chosen discipline. Similar indicators in the overall comparison were employed to benchmark at disciplinary level. International leading universities still had higher percentages of high quality papers, citation counts, and CNCI. But in some disciplines, ShanghaiTech performed better than domestic top institutions, and even came close to some international leading universities.

A representative discipline was selected to illustrate the approach. Table 3 gives the result in the area of Molecular Biology & Genetics in which almost all the institutions used for benchmarking have been active. Among all the benchmarking universities, Univ C & Univ E had better performance. We notice that the values of CNCI for Univ C and SHTech-A are 2.71 and 1.64, respectively, but SHTech-C has 2.6 in this regard, which is very close to Univ C and the international median, and significantly higher than the domestic median. This indicates that ShanghaiTech’s full-time professors in this research area are doing a better job, even when compared with its own adjunct professors who have worked in the institutes in the field and most have longer research engagement.

A content analysis was conducted to further explore the differences in research topics, using co-occurrence analysis of keywords. Here the effort was made to compare ShanghaiTech with the strongest in the chosen field. Analyses were performed resulting in a research topic map for ShanghaiTech, and another for the institution with highest CNCI in the discipline. In the WoS data, Keywords Plus extracted from titles of cited publications and footnotes provide supplementary terms to the Author’s Keywords (Chen, 2017; Garfield, 1990). Based on keywords and using the full counting method maps were generated by VOSviewer (version 1.6.10).

Univ E was selected as the leading university in the area of molecular biology & genetics. Its keyword co-occurrence map led to six main clusters (indicated with colored nodes and links, Figure 2a). The themes of these clusters are stem cell and genes (blue cluster, main keywords: gene expression, DNA methylation, human genome, embryonic stem-cells), genetic mutation (green cluster, main keywords: cell lung cancer, T cell, somatic mutations, copy-number alteration), cancer therapy mechanism (pink cluster, main keywords: cancer, stem cell, drug delivery), protein (sky blue cluster, main keywords: protein, saccharomyces-cerevisiae, crystal structure), genome research (red cluster, main keywords: cardiovascular disease, susceptibility loci, genetic architecture), medical research (yellow cluster, main keywords: human brain, Alzheimer disease, skeletal muscle). The main co-occurring (linked) keywords of each cluster indicate the main research topics in each theme.

Figure 2

In the corresponding map for ShanghaiTech, there were also six major thematic clusters, resulting from the following principal keywords: crystal structure, embryonic stem cell, gene expression, cell proliferation, cell lung cancer and hippo pathway. These six clusters have a strong linkage correlation (Figure 2b). Compared to Univ E, it seems that ShanghaiTech has narrower research coverage and the linkage connections are sparser, which might be one of the reasons of relatively lower citations and impact.

The distributions themselves should be compared cautiously. Each institution has its selected focuses in research topics so the maps will naturally differ. Moreover, in itself such a map does not provide information on the quality, impact or leading capacity. Hence, further information, such as citation impact or median year of publication of each subject, may need to be added or superimposed to the respective maps for a thorough comparison. However, looking at the overlaps or at the gaps in topic distribution serves as a first step to guide the attention onto the actual research subjects and helps identifying the strengths or uniqueness of an institution. It further alerts to missing trends or topics, and indicates for which topics new researchers are needed to fill critical gaps.

Another method to reveal the development of subjects is illustrated in Table 4. The most frequent keywords in the field of molecular biology & genetics from all benchmarking universities are listed, together with the year of first appearance during the period 2014–2018 for each word by any of the institutions, indicated by √. If, during the period 2014–2018 ShanghaiTech papers used this keyword for the first time as benchmarking universities this is indicated by grey background. It was found that for a few keywords, ShanghaiTech published papers almost at the same time with other institutions, but ShanghaiTech papers dealt with most keywords later than others. However, the time range of the data, starting at 2014, complicated the explanation because we did not know if there were occurrences earlier than 2014 for those keywords ShanghaiTech papers claimed to use concurrently with others. A much longer time range is needed.

Analyzing the contribution to research fronts can distinguish whether an institution is leading or following. A Research Front (in short RF) is a cluster of co-cited core papers and the group of current papers citing those papers. Based on the core papers associated with a RF provided in the Essential Science Indicator (ESI) database, the publication data of each university were mapped (by WoS ID). A total of 9,662

research fronts and corresponding core papers were retrieved. Then the citation counts of those core papers of each institution were obtained through WoS. ESI provides the average year of the core papers in its clusters for each RF, and to some extent it can be assumed that the earlier papers published in each RF cluster, the more likely they were leading the research topics. Based on this, the year of publication of core papers in a RF of each university was compared with average publication year of the topic in RF; we recorded if ShanghaiTech papers were published earlier/no later than the average publication year of the RF topics.

Generally, international leading universities had better performance than domestic universities. Although ShanghaiTech and Univ O had a few papers in the RF, and published some papers not later than the average publication year (Table 5), there were no papers that were published earlier than the average publication year, which shows that these two institutions are not leading in this RF.

In order to assess the research performance of internal units of ShanghaiTech (e.g. school or Principal Investigator (PI)), a graph using the number of publication, times cited, and CNCI of each PI of ShanghaiTech was obtained. The data is presented in Figure 3 with the X axis representing the number of publications for each PI; the Y axis shows the total number of citations, while node radii refer to CNCI values. Nodes, representing PIs, of different colors indicate the schools of these PIs. This figure immediately benchmarks productivity and impact for each PI of each internal unit of ShanghaiTech, with those at the right-upper corner publishing more and gaining more citations. Those at the left-lower corner published fewer and received fewer citations. Some PIs published only a few highly cited papers while other ones published more with a lower number of citations. Admittedly, this might be an overly simplified description of research productivity, but further analysis based on it may provide interesting insights.

Figure 3