1. bookAHEAD OF PRINT
Zeitschriftendaten
License
Format
Zeitschrift
Erstveröffentlichung
30 Mar 2017
Erscheinungsweise
4 Hefte pro Jahr
Sprachen
Englisch
access type Open Access

The Scientometric Measurement of Interdisciplinarity and Diversity in the Research Portfolios of Chinese Universities

Online veröffentlicht: 24 Jun 2021
Seitenbereich: -
Eingereicht: 17 Jan 2021
Akzeptiert: 01 Jun 2021
Zeitschriftendaten
License
Format
Zeitschrift
Erstveröffentlichung
30 Mar 2017
Erscheinungsweise
4 Hefte pro Jahr
Sprachen
Englisch
Introduction

On July 29, 2020, the Academic Degrees Committee of the State Council—an advisory council of the Chinese government—announced that the category “interdiscipline” had been added to the list of national disciplines accessible for academic degrees. This initiative will not only result in a structural change to China's classification of academic degrees, it was also designed to promote the future development of interdisciplinarity in China. As a case in point, three months after its release in late October 2020, the National Natural Sciences Foundations of China (NSFC) announced the launch of a new department for “interdisciplinary studies”. This will be the ninth department of the NSFC, and will focus on funding interdisciplinary projects. As the first change to the NSFC funding scheme in 11 years, the decision has drawn much attention.

Interdisciplinarity is a hot topic in science and technology policy. However, the concept of interdisciplinarity is both abstract and complex, which makes it difficult to fully represent or measure interdisciplinarity in terms of indicators, which can be compared among them. A variety of measures for diversity, as a proxy of interdisciplinarity, has been proposed in the literature. Further, one can find such indicators to measure the interdisciplinarity of a set of articles, patents, or journals. In this study, we ask: Can one rank institutions in terms of their disciplinary diversity? And, if so, what does this tell us about interdisciplinarity?—noting that diversity is not necessarily a goal universities strive for; some aspire to be the best in a particular discipline.

During the last few years, we, the authors of this paper, have explored the scientometric measurement of interdisciplinarity and diversity in scholarly communications in collaboration with a number of colleagues. Contributions to this program of studies were made (in alphabetic order) by Lutz Bornmann, Wolfgang Glänzel, Inga Ivanova, Ronald Rousseau, Caroline S. Wagner, and Ping Zhou (Leydesdorff & Ivanova, 2020; Leydesdorff, Wagner, & Bornmann, 2018 and 2019; Zhang, Rousseau, & Glänzel, 2016; Zhang, Sun, Chinchilla-Rodríguez, Chen, & Huang, 2018; Zhang, Sun, Jiang, & Huang, 2021). One of our objectives has been to develop a non-commercial, public-domain application that allows researchers and policy analysts to measure the diversity of any document set or network structure using a range of indicators. To our best knowledge, no such tool has ever been developed, at least not for public consumption.

A large number of indicators of “diversity” have been proposed in the literature (e.g. Rao-Stirling diversity; Stirling (2007), the Gini-coefficient, Simpson (1949) indicator, Hirschman-Herfindahl (Herfindahl, 1950; Hirschman, 1945), etc. In this communication, we report on the facilities which we created during the last two years. Particularly, we introduce the freely available program interd_vb.exe (available at http://www.leydesdorff.net/software/interdisc.2020/) for this purpose. We document the various options and provide instructions for practitioners interested in measuring diversity and interdisciplinarity. By elaborating on the measurement of the disciplinary diversity of the research portfolios of the 42 top universities listed as the “Double First-Class” universities (Liu et al., 2018), we are able to show the options and choices to be made given the current state of the art.

Technical instructions are additionally available at http://www.leydesdorff.net/software/interdisc.2020/index.htm. The inputs and outputs are in .csv format. The same output is also stored in interdis.bdf. The subsequent analysis demonstrates the options and choices that can be made as route to a final comparison. As a disclaimer, note that we are in no way professional programmers. We cannot guarantee that our routines are error-free, and we acknowledge that the user interface could be improved. However, as a test, one of us programmed the application in two different computer languages, and the results were virtually the same. Additionally, we do believe the functionality is unique and, therefore, state of the art for what it is.

One of the advantages of the application is its ability to handle large volumes of data. For example, the need to analyze an entire database, such as Web-of-Science (WoS), Scopus, or Google Scholar, is becoming increasingly common. Analyses of this magnitude can generate baselines for evaluating the disciplinary diversity of articles, journals, topics, etc. The Interdisc program can relieve the computational overhead of processing massive amounts of data. That said, although the equations used to calculate diversity indicators are often mathematically transparent, specifying the terms as computer code can help analysts to further precision in decisions that would not otherwise be involved in a manual calculation.

The relation of the indicators to bibliometrics

Interdisciplinarity can be operationalized as references to different literatures. Such co-citing is known in scientometrics as bibliographic coupling (Kessler, 1963). When a document, for example, cites both articles in physics journals and in sociology journals, this can be expected to indicate interdisciplinarity more than citing chemical physics and solid-state physics in the same document or in the same set. In other words, one couples literature from different disciplines in the references. This coupling can be at the level of articles, journals, or Web-of-Science Subject Categories (WCs).

Bibliographic coupling is an indicator on the citing side and thus the operation opposite to co-citation: co-citations across disciplinary borders indicate interdisciplinary diffusion, whereas the measurement of interdisciplinarity by bibliographic coupling focuses on aggregated citing behaviour.

Whereas “interdisciplinarity” by citing papers refers to documents, documents are often not the units of analysis in the case of research evaluation at the institutional level. The interdisciplinary operator of bibliographic coupling is defined in terms of disciplines and not in terms of institutions. Does the diversity of a university in terms of departments indicate interdisciplinarity or only comprehensiveness of a research portfolio? Since there is no coupling in terms of different fields, one may measure only comprehensiveness, and not interdisciplinarity.

Institutional units are primarily administratively and not disciplinarily organized. The diversity indicators apply to disciplinary differentiations; social differentiation in terms of departments, etc., may have a different meaning. For example, diversity may also indicate comprehensiveness. How does this work out empirically?

Indicators of diversity

In this section, we first discuss the following indicators of diversity and interdisciplinarity in terms of the basic equations:

Shannon's entropy

Using Shannon's (1948) information theory, one can measure diversity as the uncertainty in a distribution. The equation of the Shannon entropy can be stated as follows: H=pilog(pi)H = - \sum {{p_i}\log \left( {{p_i}} \right)} Where = pi = xi / X, and ∑pi = 1. xi denotes the number of cells belonging to subject category i. Based on information theory, the maximum capacity (Hmax) of a system is composed of two parts which are (1) the number of realized states and (2) the not-yet-realized but possible states (HmaxHsystem); that is, the redundancy. Leydesdorff and Ivanova (2021t) proposed to use redundancy as a measure of synergy.

The Simpson Index

The Simpson index was originally developed to measure “concentration” (Rousseau, 2018; Simpson, 1949). Stirling (2007) introduced the concept into the field of scientometrics as a way to evaluate the variety of subject categories and the unevenness in the distribution of these categories. For this reason, Simpson diversity is often called a “dual concept” indicator of diversity. It combines variety with balance in a single number. The equation for Simpson's diversity index is SI=1pi2SI = 1 - \sum {p_i^2} where pi = xi / X, X = ∑xi, and xi denotes the number of elements belonging to the subject category i.

Rao-Stirling index

Stirling (2007) proposed Rao-Stirling (RS) diversity to measure interdisciplinarity, distinguishing variety, balance, and disparity as the three components of interdisciplinarity. Formally, the indicator is calculated as RS=i,j(pipj)αdijβRS = {\sum\nolimits_{i,j} {{{\left( {{p_i}{p_j}} \right)}^\alpha }d} _{ij}}^\beta where dij (or equivalently 1-Sij) denotes the distance between subject i and subject j, and Sij is the similarity between the subjects i and j. pi = xi / X, X = ∑xi, and xi denotes the number of cells belonging to subject i. The exponents α and β are two parameters for adjusting the relative weights of distance dij and variety or balance pipj.

The novelty of RS lies in the disparity term (dij). The other part of Eq. 3 is the same as the Simpson index, which measures both variety and balance.

In most scientometric applications, α and β are set to 1 (Rafols & Meyer, 2010), which simplifies Eq. (3) to: D=ijdijpipjD = \sum\nolimits_{i \ne j} {{d_{ij}}{p_i}{p_j}}

True RS diversity

True RS diversity has its origins in a variant of the Hill indicator proposed by Leinster and Cobbold (2012) which adds disparity into the Hill equation traditionally used in ecology. This indicator was subsequently modified by Zhang et al. (2016) as follows: 2DS=(i=1npi(j=1nsijpj))1=1i,j=1nsijpipj^2{D^S} = {\left( {\sum\limits_{i = 1}^n {{p_i}\left( {\sum\limits_{j = 1}^n {{s_{ij}}{p_j}} } \right)} } \right)^{ - 1}} = {1 \over {\sum\limits_{i,j = 1}^n {{s_{ij}}{p_i}{p_j}} }} where Sij denotes the similarity between subjects i and j. pi = xi / X, X = ∑xi, and xi is the number of cells belonging to subject i. Note that True RS is no longer bounded between zero and one, and it allows the parameters to be scaled such that one unit of study is, say, twice as interdisciplinary as another.

DIV

Stirling (1998) stated that “any integration of variety and balance into dual concept diversity must necessarily involve the implicit or explicit prioritization of the subordinate properties”. From this, Leydesdorff et al. (2019) proposed a new diversity indicator, called DIV, that divides interdisciplinarity into its three components (variety, balance, and disparity) and recombines them by multiplication. An empirical experiment proves the advantages of this new indicator over RS diversity. Formally, DIV is expressed as follows: DIVc=[ncN]*[1G(c)]*[j=nci=nci=1,j=1,ijdij{nc*(nc1)}]{DIV_c} = \left[ {{{{n_c}} \over N}} \right]*\left[ {1 - G\left( c \right)} \right]*\left[ {\sum {\matrix{ {j = {n_c}} \hfill \cr {i = {n_c}} \hfill \cr {i = 1,} \hfill \cr {j = 1,} \hfill \cr {i \ne j} \hfill \cr } } {{{d_{ij}}} \over {\left\{ {{n_c}*\left( {{n_c} - 1} \right)} \right\}}}} \right] where n(c) is the number of elements in the case under study; N is the total number of elements in the set; c is the sequence number of the column vector in the set; G(c) is the Gini coefficient of c; and dij is the level of disparity between elements i and j.

Rousseau (2019) suggested some improvements to DIV. He showed that DIV can be turned into a measure of True Diversity by removing the term N (variety) in the denominator of Eq. 6. Rousseau argued that a better framework for diversity measurement would account for several requirements, not all of which are met by existing frameworks. Responding to the improvements made by Rousseau (2019), Leydesdorff, Wagner, and Bornmann (2019) provided an updated version of the improved DIV* as a True Diversity measure: DIVc*=nc*[1G(c)]*[j=nci=nci=1,j=1,ijdij{nc*(nc1)}]DIV_c^* = {n_c}*\left[ {1 - G\left( c \right)} \right]*\left[ {\sum {\matrix{ {j = {n_c}} \hfill \cr {i = {n_c}} \hfill \cr {i = 1,} \hfill \cr {j = 1,} \hfill \cr {i \ne j} \hfill \cr } } {{{d_{ij}}} \over {\left\{ {{n_c}*\left( {{n_c} - 1} \right)} \right\}}}} \right] where n(c) is the number of elements in subject c; G(c) is the Gini coefficient of c; and dij is the level of disparity between elements i and j.

Gini coefficient

The Gini coefficient is a well-known indicator for representing income inequality among people and wealth inequality among nations (Lorenz, 1905). Hence, when measuring the diversity of interdisciplinary research with the Gini coefficient, the research is treated as a system comprised of three elements—variety, balance, and disparity (Porter & Rafols, 2009; Rafols & Meyer, 2010) where (1 – Gini) is used as the indicator of balance (Nijssen et al., 1998).

The theory of relative mean differences defines the Gini coefficient as (e.g. Buchan, 2002): G=i=1nj=1n|xixj|2n2x¯G = {{\sum\nolimits_{i = 1}^n {\sum\nolimits_{j = 1}^n {\left| {{x_i} - {x_j}} \right|} } } \over {2{n^2}\bar x}} where x is an observed value, n is the number of values observed, and x bar is the mean value.

Note, however, that there are several alternative definitions of the Gini coefficient. See, for example, that provided at https://en.wikipedia.org/wiki/Gini_coefficient (cf. Rousseau (1992)).

If the x values are first placed in ascending order such that each x has rank i, some of the comparisons above can be avoided and computation is therefore more efficient, i.e.: G=2n2x¯i=1ni(xix¯)G = {2 \over {{n^2}\bar x}}\sum\limits_{i = 1}^n {i\left( {{x_i} - \bar x} \right)} G=i=1n(2in1)xini=1nxiG = {{\sum\nolimits_{i = 1}^n {\left( {2i - n - 1} \right){x_i}} } \over {n\sum\nolimits_{i = 1}^n {{x_i}} }} where x is an observed value, n is the number of values observed, and i is the rank of values in ascending order.

For G to be an unbiased estimate of the true population value, it should be multiplied by n/(n-1) (Dixon, 1987; Mills & Zandvakili, 1997). In the bibliometric literature, this index is also known as the Pratt index (Pratt, 1977). The value of both the Gini and the normalized G are provided by interd_vb.exe.

Other indicators

The concept of coherence based on network analysis has attracted attention from researchers in scientometrics (e.g. Rafols, 2014). While the diversity indicators rely on a pre-defined category system, coherence can be generated via a bottom-up approach that describes the intensity of the relations between any elements in a network. From this perspective, comprehensive frameworks composed of diversity and coherence have been proposed to improve the depiction of interdisciplinary systems (Rafols & Meyer, 2010).

The computation of diversity and interdisciplinarity indicators

The program interd_vb.exe (http://www.leydesdorff.net/software/interdisc.2020/interd_vb.exe) was rewritten based on the routine Mode2Div.exe previously programmed in the so-called xBase language. Unfortunately, computing cosine values for large matrices can be time-consuming with xBase, which imposes a soft limit on the size of the datasets that can be processed. Hence, we rewrote Mode2Div. exe in Visual Basic 6 to become interd_vb.exe, i.e. the online Interdisc application. Visual Basic 6 runs on Win10 (32/64 bits) and does not require the predetermined amount of memory to be allocated to processing. Therefore, the only limitation to the size of the dataset that can be processed is hardware. The two programs, interd_vb.exe and Mode2Div.exe, have similar objectives but a different organization and architecture, and the results they produce are exactly the same. Both programs are documented in Leydesdorff et al. (2018, 2019) and the software is available for download from https://www.leydesdorff.net/software/interdisc.2020/ and Figshare (https://figshare.com/account/articles/12871529).

One key difference between the two versions of the program is their input requirements. In the case of mode2div.exe, the input is stored listwise using the Pajek format, each line describing the row and column of a cell in a matrix of values. Thus, the input can be read as three fields without any system limitations. The data is assumed to be 2-mode so that an asymmetrical (citation) matrix can be processed. The program then computes the diversity measures along the column vectors of a data matrix saved in .csv format. As an example, to measure the interdisciplinarity of a set of documents, one could use jcitnetw.exe

https://www.leydesdorff.net/software/interdisc/jcitnetw.exe

to easily generate a co-occurrence matrix of cited journals in the Pajek format, using plain text downloaded from the Web of Science. More details on this can be found at https://www.leydesdorff.net/software/mode2div/.

The distance metric and the disparity measure

Stirling (2007) added a new element to diversity measurement: disparity. Disparity indicates the distance between two subjects in the sample(s) under study. For example, if the distances in a subset are small, this space can be considered a niche of related variety (Frenken et al., 2007). However, disparity as a factor in both RS and the DIV requires the choice of a distance metric. Following Salton and McGill (1983), Ahlgren, Jarneving, and Rousseau (2003) proposed cosine as a non-parametric measure of similarity for bibliometrics. From a comparison of a number of similarity/distance measures, Egghe and Leydesdorff (2009) concluded that the cosine fulfills a number of requirements.

Like Pearson correlations, cosine values are defined in a vector space and are therefore positional, whereas the very similar Jaccard index is relational. Unlike the Pearson correlation, however, cosines do not normalize to a mean and, since bibliometric distributions are highly skewed, normalizations using the mean are to be avoided. Our routines use (1 – cosine), which can be considered a distance measure. Pragmatically, the terms of a cosine can be written as co-occurrence in the numerator and the sum of squares along the two column vectors x and y multiplied in the denominator. Note that, here, the matrix rows contain the disciplines and the columns contain the universities, so the cosine values are computed between the row vectors.

One disadvantage of Mode2Div.exe is that data is often not readily available in Pajek format and converting the data into this format may generate other problems (Pfeffer, Mrvar, & Batagelj, 2013). The most generic format for data, however, is a matrix as a comma or tab-separated plain ASCII file. There are no size limitations for this data, although Excel (depending on the Office version) may not allow for more than 255 variables. This data, however, can also be written using a text editor (e.g. the freeware Note++) or any other program. The size of the matrix is only limited by external factors such as free diskspace.

The routine begins with asking for the name of the .csv file containing the variables and the number of vectors to be compared for the purposes of error correction. The file is then rewritten into output which is reported in the files interdis.dbf and equivalently interdis.csv. The specific differences in terms of inputs, outputs, and other related items about these programs are summarized in AppendixTable S1.

See for further details at http://www.leydesdorff.net/software/interdisc.2020/

Data

As empirical data, we used the portfolio of research articles from the 42 Chinese universities listed as “Double First-Class universities” between 2017 (when the list was first released) and 2019. The Chinese government offers substantial support to this select group of universities through a series of special programs. Additionally, although this particular list has only been published since 2017, similar initiatives under different names have existed periodically since the 1990s, with the majority of universities considered to be elite remaining much the same this whole time. Thus, these 42 institutions were selected because this group is both clearly delineated and large enough to provide a large-scale sample. In addition, we also included the portfolios of two well-known American universities, Harvard and Stanford, to provide a standard those in the West might find easier to benchmark. In a subsequent article, Leydesdorff, Wagner, and Zhang (2021), we further compare these results with 205 Chinese universities.

Each of the universities in the sample promotes itself as a comprehensive university. However, some note specific missions or strengths; for instance, the agricultural universities. The publications associated with each university were retrieved using the organization's name and/or its variants from the Preferred Organization Index in WoS.

The domains searched include the Science Citation Index Expanded (SCI-E), the Social Sciences Citation Index (SSCI), and the Arts & Humanities Citation Index (A&HCI) in the Web of Science (WoS) Core Collection. We limited the document type to articles and reviews. The number of articles retrieved per university are listed in Table 1 in decreasing order.

Number of publications associated with the 44 universities in our sample (2017–2019); in decreasing order.

No.University namePapersNo.University namePapers
1Harvard Univ76,14423Northeastern Univ14,893
2Shanghai Jiao Tong Univ37,01624Beihang Univ14,484
3Zhejiang Univ35,20425Dalian Univ of Technology13,861
4Tsinghua Univ32,68126Zhengzhou Univ12,993
5Stanford Univ32,42827Northwestern Polytechnical Univ12,497
6Peking Univ30,16028Chongqing Univ12,451
7Sun Yat-Sen Univ26,82329Univ of Electronic S & T of China12,334
8Huazhong Univ of S & T24,82230Xiamen Univ11,607
9Fudan Univ24,47531Beijing Institute of Technology11,206
10Sichuan Univ23,25932Beijing Normal Univ10,043
11Central South Univ22,87033Nankai Univ9970
12Xi’an Jiaotong Univ22,69834Hunan Univ9811
13Shandong Univ21,60135Lanzhou Univ9156
14Jilin Univ21,06836China Agricultural Univ8762
15Harbin Institute of Technology20,75037Northwest A & F Univ7817
16Univ of S & T of China20,74738East China Normal Univ7610
17Wuhan Univ19,74839National Univ of Defense Technology6601
18Nanjing Univ19,24640Ocean Univ of China6390
19Tianjin Univ17,77841Renmin Univ of China2946
20Tongji Univ17,22642Yunnan Univ2835
21Southeast Univ16,95943Xinjiang Univ1979
22South China Univ of Technology15,59544Minzu Univ of China760

We first organized the data into an asymmetrical occurrence matrix of the 44 universities against 254 WoS categories. We then computed the six diversity measures using Interd_vb.exe.

Results
Ranking of universities in terms of interdisciplinarity

The interdisciplinarity scores for each indicator and university are listed in Table 2. Additionally, we have provided a ranking against each indicator. For example, for the DIV* indicator, Stanford University is ranked No. 1, whereas, according to the True RS indicator, it is ranked No. 15. Tsinghua University, which is widely considered to be the top university in China, sits in 21st place on the list of DIV*. Keep in mind, however, that this is a ranking of comprehensiveness as measured by disciplinary diversity, not of impact. As mentioned in Section 2.6, the Gini coefficient is a measure of unbalance, and therefore (1 – Gini) is used in the computation of DIV* (Eq. 7; Table 2).

The Indicator scores generated by interd_vb.exe routine.

UniversityDIV*RankTrue RSRankSimpsonRankShannonRankVarietyRankDisparityRank(1-Gini)Rank
Stanford Univ40.26011.503150.98616.83110.98810.472230.3401
Sun Yat-Sen Univ35.75421.54960.98356.66320.94550.474120.3142
Peking Univ33.35231.516130.98276.56840.95330.474150.2914
Zhejiang Univ33.23741.54970.98336.59430.94940.473180.2923
Harvard Univ32.32851.288390.98366.51270.98810.471250.2747
Shanghai Jiao Tong Univ31.15161.55350.98426.56550.92190.471260.2835
Sichuan Univ30.09271.527120.98346.51760.913110.473190.2746
Wuhan Univ29.11781.54880.98296.46580.917100.473160.2648
Northeastern Univ28.89291.485180.975246.335150.94550.466370.25810
Fudan Univ28.102101.457220.979176.361120.92970.468350.25412
Shandong Univ27.683111.492170.981116.41690.898130.472240.25711
East China Normal Univ27.471121.495160.980126.415100.886180.470280.2609
Nanjing Univ26.735131.444240.977206.256210.92970.47590.23819
Beijing Normal Univ26.427141.57540.976226.328160.890160.467360.25013
Xiamen Univ26.392151.439250.977186.301190.894140.474130.24516
Tongji Univ26.286161.538110.979136.341130.894140.471270.24615
Huazhong Univ of S&T25.700171.452230.979156.308180.886180.475100.24118
Central South Univ25.298181.54590.979166.336140.843230.47920.24714
Lanzhou Univ23.959191.513140.981106.362110.815260.472220.24517
Jilin Univ23.323201.431260.975266.177220.866210.474140.22422
Tsinghua Univ23.253211.371290.975276.120250.913110.470290.21325
Xi’an Jiaotong Univ22.879221.409270.975256.128240.890160.472210.21424
Zhengzhou Univ21.553231.463200.976236.152230.811270.47570.22023
Southeast Univ20.325241.385280.970335.971290.870200.470300.19628
Renmin Univ20.323251.458210.979146.277200.748350.457410.23420
Yunnan Univ18.896261.540100.98286.314170.681390.469310.23321
Nankai Univ18.091271.334320.969405.894320.827240.462400.18730
Univ of S&T – China17.782281.303370.968415.797380.850220.47750.17238
Tianjin Univ17.466291.296380.970355.852370.819250.475110.17735
South China Univ of Technol17.286301.349310.970365.882330.795280.473200.18131
Chongqing Univ17.029311.313340.970345.856360.795280.47830.17636
Hunan Univ16.958321.307360.973305.913310.772320.48010.18032
Ocean Univ of China16.824331.73410.977216.102260.677400.473170.20726
Dalian Univ of Technol16.509341.315330.974295.922300.756340.47840.18033
Harbin Inst of Technology15.412351.286400.969385.769390.776310.47580.16539
Beihang Univ15.007361.311350.970375.762400.780300.465380.16340
China Agricultural Univ14.671371.67030.973315.873350.701370.469340.17637
Northwest A&F Univ14.040381.68120.972325.881340.665410.469330.17734
Beijing Inst of Technol13.944391.269440.969395.728410.724360.47560.15941
Xinjiang Univ12.921401.369300.975285.997280.571420.463390.19329
Univ of Electronic S&T of China12.847411.281420.950445.428430.768330.448430.14742
Minzu Univ of China12.104421.464190.977196.049270.535440.448420.19927
Northwestern Polytechnical Univ12.062431.275430.962425.571420.693380.469320.14643
National Univ of Defense Technol7.783441.285410.951435.274440.563430.446440.12244

The Spearman rank-order correlations are provided in Table 3. The DIV* indicator correlates much more closely to the VARIETY and GINI indicator, as is to be expected since (1-GINI) is actually used to calculate DIV.* H owever, there is only a moderate correlation between the two true diversity indicators, True RS and DIV* at (ρ = 0.50; p < 0.01). Further, the rankings of the top five universities according to these two indicators are inconsistent. These unexpected results raise further questions.

Spearman's correlations for ranking order generated by Interd_vb.exe (N = 42).

DIV*TRUE RSVARIETYDISPARITY(1 -GINI)SIMPSON SHANNON
DIV*
TRUE RS.563**
VARIETY.926**.323*
DISPARITY.215−.092.230
(1 – GINI).936.717**.772**.074
SIMPSON.789**.766**.551**.085.917**
SHANNON.911**.734**.725**087.990**.950**

Correlation is significant at the 0.01 level (2-tailed).

Correlation is significant at the 0.05 level (2-tailed).

The new element added to the Striling (2007) to the measurement of diversity and interdisciplinarity was disparity. In Table 3, disparity indeed is not significantly correlated with any of the other diversity indicators. Factor analysis of this data (Table 4) shows disparity (and variety) as a second component. Unlike True RS, DIV* captures both dimensions, as was Stirling's theoretical intention.

Factor analysis of the interdisciplinarity and diversity indicators (N = 42).

Rotated Component Matrixa
Component
12
True RS.881−.133
Shannon.877.455
(1-Gini).862.456
Simpson.830.390
Div*.703.657
Variety.329.853
Disparity.792

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 3 iterations; 85.1% of the variance explained.

As stated above, when applying interd_vb.exe, the terms of the cosine are pragmatically computed using co-occurrences in the sample in the numerator and the square roots of the products of sum of squares along the thus affiliated vectors x and y in the denominator. Disparity is then defined as the sum of local values of (1-cosine) over the set. This matrix is a “sample-dependent” local matrix since it reflects the disparity within the data samples. Consequently, these values vary with the data-sample used as input. It may often be convenient for analysts and developers to calculate the diversity values in this way (locally), particularly, when one has no access to a global disparity matrix. However, the systems of reference for the cosine-normalization are then different among samples.

Local versus global disparity

In contrast to local disparity, using a global matrix solves (almost by definition) the problem of comparability across samples. To demonstrate the difference between “local” and “global” matrices, we recalculated the diversity scores using a global cosine matrix based on the full set of JCR data for 2019. These data include 236 subject categories in the Science and Social Sciences Citation Indexes (but not the 25 in the Arts & Humanities Citation Index).

The results for both DIV* and True RS are shown in Table 5, and Table 6 shows the Spearman's correlations for the ranking order of the two indicators.

The local cosine matrix was generated with interdisc_vb.exe; the global one was retrieved from http://www.leydesdorff.net/software/wc19. The cosine similarity matrix for the WoS categories based on JCR 2019 data is also provided at http://www.leydesdorff.net/wc15/wc19.

As expected, the correlation between DIV* and True RS (or RS) increased (from 0.502 to 0.695), demonstrating that the consistency between different diversity indicator values can be improved by using a global matrix instead of a local matrix.

Local vs. global disparity using JCR data for 2019.

UniversityDIV*RankTRUE RSRank
Stanford Univ72.95615.4882
Sun Yat-Sen Univ68.42924.7415
Zhejiang Univ63.34334.30012
Peking Univ62.65444.6328
Shanghai Jiao Tong Univ60.90754.6437
Sichuan Univ58.36764.03319
Harvard Univ58.30174.4619
Wuhan Univ56.90384.7026
Northeastern Univ54.88794.16115
Fudan Univ54.196103.89722
Shandong Univ53.921114.16214
East China Normal Univ51.757124.30911
Xiamen Univ51.354133.70524
Tongji Univ51.348144.8234
Beijing Normal Univ50.815155.0823
Huazhong Univ of S&T50.632163.96320
Central South Univ50.535174.10717
Nanjing Univ50.285183.85123
Lanzhou Univ47.622194.08718
Jilin Univ46.049203.29234
Xi’an Jiaotong Univ45.655213.66425
Tsinghua Univ45.121223.60126
Zhengzhou Univ43.389233.44229
Southeast Univ39.662243.90221
Renmin Univ37.896255.5631
Nankai Univ36.427262.95043
Yunnan Univ36.236274.38210
Univ of S & T – China35.002282.87644
Tianjin Univ34.613292.99541
South China Univ of Technol34.388302.97842
Ocean Univ of China32.747314.20213
Chongqing Univ32.519323.26035
Hunan Univ32.394333.37932
Dalian Univ of Technol31.933343.35533
Harbin Inst of Technol30.166353.19136
Beihang Univ30.029363.50828
China Agricultural Univ29.258373.39631
Northwest A & F Univ27.904383.40230
Beijing Inst of Technol27.102393.18437
Univ of Electronic S&T of China26.892403.07339
Xinjiang Univ25.828413.53127
Northwestern Polytechnical Univ23.873423.03140
Minzu Univ of China22.645434.13216
National Univ of Defense Technol16.236443.11838

Spearman's correlations for consistency of rank order – local vs. global disparity.

DIV*_localTRUE RS_localDIV*_globalTRUE RS_global
DIV*_local
TRUE RS_local.502**
DIV*_global.996**.516**
TRUE RS_global.697**.707**.695**

Correlation is significant at the 0.01 level (2-tailed).

With a correlation between the local and global values of DIV* at .996, DIV* is obviously not sensitive to the scaling. As Rousseau (2019) noted, the disparity in DIV* “is just a relative (normalized) sum.” With hindsight, this seems an advantage of DIV* when compared with True RS.

Differences among specific universities

There are some interesting observations to be made in terms of the results of specific universities. Comparing Stanford University and Tsinghua University as examples, Stanford University ranks significantly higher than Tsinghua according to both DIV* and True RS, as shown in Table 4. The science overlay maps in Figures 1 and 2 illustrate this vividly (Carley et al., 2017; Leydesdorff et al., 2016; Rafols et al., 2010). Using VOS Viewer for the visualization (Waltman et al., 2010), each node represents a WoS category, and the size of the node indicates the number of publications.

Figure 1

Science overlay map of the publications with an address at Tsinghua University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).

Figure 2

The science overlay map of the publications associated with Stanford University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).

It is clear (on the basis of visual inspection of these two maps) that the category distributions of the two universities are very different. Stanford University obviously prioritizes research in Clinical Medicine, Biomedicine, and other medical disciplines, while Tsinghua University has a clear focus on Computer Science & Engineering, Material Science, and other Engineering fields. However, although each university has strengths in particular disciplines, the distribution of disciplines across Stanford's portfolio is more balanced than that across Tsinghua's.

Discussion and conclusion

DIV* values were more in line with our intuition about the diversity of these universities than the RS or True RS values. The latter, particularly worsen when the results are based on local disparity matrices. Using this local matrix, however, some field-specific universities like Ocean University of China and the Northwest Agriculture & Forestry University are found to have high diversity values with the True RS (and RS) indicators. These results raise further questions.

The results for RS/True RS are more sensitive than DIV* to the choice of similarity measures (Rafols & Leydesdorff, 2010). As Rousseau (2019) notes: “DIV, taking disparity into account as just a relative (normalized) sum” is not sensitive to scaling. In Eq. (8), disparity is only defined at the level of the sample; the interaction between category i and category j (pi and pj, respectively) with dij is not taken into account at the cell level, only the total sum of all disparity values is.

Table 2 (above) showed that the Ocean University has the highest True RS diversity of all universities. However, when checking the specific distribution of Web of Science categories, we found that more papers are published within Oceanography (14.01%) than any other category. Yet, Oceanography is a relatively marginal category in our sample, with much lower cosine similarities than other categories. As a result, the disparity (1-cosine) between Oceanography and other categories is much higher than on average, at a value of 0.73 vs 0.47, respectively. The extraordinarily high proportion of publications in Oceanography and the category's high disparity from other categories leads to an unexpectedly high diversity value when measured with RS/True RS. However, when using a global similarity matrix (Table 4), the scores of RS/True RS in most field-specialized universities decreased. As noted, these rankings were not affected by this effect when using DIV*.

The portfolio of papers with a Harvard address covers a wide range of categories and the distribution is relatively balanced. However, the cosine similarities of the categories with most publications are relatively high, i.e. they tend to have low disparity values, which results in a lower valus of RS/True RS when using a local similarity matrix. These empirical results suggest that RS diversity values based on a global disparity matrix provide results that are more in line with expectations. Therefore, insofar as a user has access to a global matrix one is advised to use this instead of the values generated endogenously by our software.

When universities operate in similar markets with the same institutional imperatives, such as tasks specified in national legislation, one might expect them to develop isomorphism (Halffman & Leydesdorff, 2010; Powell & DiMaggio, 1991; Wagner, Bornmann, Cai, & Leydesdorff, in preparation). However, our results indicate that universities do not tend toward isomorphism when it comes to comprehensiveness, as they do with impact. We reason that this is because impact is measured and prioritized in the bureaucratic frameworks of the state, whereas comprehensiveness is influenced by local opportunities, such as emerging technologies in the companies geographically or intellectually nearby. Hence, developing a deeper understanding of institutional comprehensiveness demands consideration of a broader context and more aspects of society, such as missions of specific universities.

Our analysis clarifies further differences between impact and comprehensiveness. Competition for impact pertains to quality, while competition for diversity/specialty pertains to differentiation. For example, shielding intellectual property rights is specific to a university's relations with industry. When it comes to comprehensiveness, the specificity of the knowledge content matters more than the formal criteria of measuring and comparing output and impact. In our opinion, interdisciplinarity, diversity, or comprehensiveness should not be considered another type of impact. While impact can be formalized across units of operation, e.g. faculties, departments, etc., after proper normalization, diversity or comprehensiveness remains content-based.

In other words, the analytical distinction between intellectual and social organization does not mean that the two dimensions can be traded off at the level of a university. On the contrary, one can expect a correlation, whether positive or negative, between the different types of research efforts. However, the differences between the two make it urgent that we develop a set of indicators for measuring diversity comparable to those of impact. By making an application available that allows users to generate the various measures of diversity for any data matrix, we hope to have contributed to this objective of quantifying and measuring diversity.

Finally, we note that although diversity is often used as a proxy for measuring interdisciplinarity, one should not expect any simplistic index to produce an informative outcome on its own (Abramo et al., 2018). The interpretations of the values of indicators should always be addressed according to the context, the purpose, and the specific object under study. The empirical analysis of the 42+ Chinese universities in terms of diversity measures not only relates to interdisciplinarity at the intellectual level, but also reflects comprehensiveness at the institutional level. Although comprehensiveness is not necessarily a goal of universities, it may reflect the status quo of disciplinary diversity within a university (or at least the structural feature of a disciplinary distribution). The measurement results of this study provide a knowledge base for understanding portfolios. A better understanding may provide new windows on potential policies and thus facilitate the development of interdisciplinarity within a university.

Figure 1

Science overlay map of the publications with an address at Tsinghua University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).
Science overlay map of the publications with an address at Tsinghua University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).

Figure 2

The science overlay map of the publications associated with Stanford University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).
The science overlay map of the publications associated with Stanford University. [Note: The base map of disciplines was developed from the matrix of 227 × 227 cells of WoS categories. This was generated on the basis of direct citation counting and normalized with the cosine function (Carley et al. 2017).

Number of publications associated with the 44 universities in our sample (2017–2019); in decreasing order.

No.University namePapersNo.University namePapers
1Harvard Univ76,14423Northeastern Univ14,893
2Shanghai Jiao Tong Univ37,01624Beihang Univ14,484
3Zhejiang Univ35,20425Dalian Univ of Technology13,861
4Tsinghua Univ32,68126Zhengzhou Univ12,993
5Stanford Univ32,42827Northwestern Polytechnical Univ12,497
6Peking Univ30,16028Chongqing Univ12,451
7Sun Yat-Sen Univ26,82329Univ of Electronic S & T of China12,334
8Huazhong Univ of S & T24,82230Xiamen Univ11,607
9Fudan Univ24,47531Beijing Institute of Technology11,206
10Sichuan Univ23,25932Beijing Normal Univ10,043
11Central South Univ22,87033Nankai Univ9970
12Xi’an Jiaotong Univ22,69834Hunan Univ9811
13Shandong Univ21,60135Lanzhou Univ9156
14Jilin Univ21,06836China Agricultural Univ8762
15Harbin Institute of Technology20,75037Northwest A & F Univ7817
16Univ of S & T of China20,74738East China Normal Univ7610
17Wuhan Univ19,74839National Univ of Defense Technology6601
18Nanjing Univ19,24640Ocean Univ of China6390
19Tianjin Univ17,77841Renmin Univ of China2946
20Tongji Univ17,22642Yunnan Univ2835
21Southeast Univ16,95943Xinjiang Univ1979
22South China Univ of Technology15,59544Minzu Univ of China760

Factor analysis of the interdisciplinarity and diversity indicators (N = 42).

Rotated Component Matrixa
Component
12
True RS.881−.133
Shannon.877.455
(1-Gini).862.456
Simpson.830.390
Div*.703.657
Variety.329.853
Disparity.792

The Indicator scores generated by interd.vb.exe routine.

UniversityDIV*RankTrue RSRankSimpsonRankShannonRankVarietyRankDisparityRank(1-Gini)Rank
Stanford Univ40.26011.503150.98616.83110.98810.472230.3401
Sun Yat-Sen Univ35.75421.54960.98356.66320.94550.474120.3142
Peking Univ33.35231.516130.98276.56840.95330.474150.2914
Zhejiang Univ33.23741.54970.98336.59430.94940.473180.2923
Harvard Univ32.32851.288390.98366.51270.98810.471250.2747
Shanghai Jiao Tong Univ31.15161.55350.98426.56550.92190.471260.2835
Sichuan Univ30.09271.527120.98346.51760.913110.473190.2746
Wuhan Univ29.11781.54880.98296.46580.917100.473160.2648
Northeastern Univ28.89291.485180.975246.335150.94550.466370.25810
Fudan Univ28.102101.457220.979176.361120.92970.468350.25412
Shandong Univ27.683111.492170.981116.41690.898130.472240.25711
East China Normal Univ27.471121.495160.980126.415100.886180.470280.2609
Nanjing Univ26.735131.444240.977206.256210.92970.47590.23819
Beijing Normal Univ26.427141.57540.976226.328160.890160.467360.25013
Xiamen Univ26.392151.439250.977186.301190.894140.474130.24516
Tongji Univ26.286161.538110.979136.341130.894140.471270.24615
Huazhong Univ of S&T25.700171.452230.979156.308180.886180.475100.24118
Central South Univ25.298181.54590.979166.336140.843230.47920.24714
Lanzhou Univ23.959191.513140.981106.362110.815260.472220.24517
Jilin Univ23.323201.431260.975266.177220.866210.474140.22422
Tsinghua Univ23.253211.371290.975276.120250.913110.470290.21325
Xi’an Jiaotong Univ22.879221.409270.975256.128240.890160.472210.21424
Zhengzhou Univ21.553231.463200.976236.152230.811270.47570.22023
Southeast Univ20.325241.385280.970335.971290.870200.470300.19628
Renmin Univ20.323251.458210.979146.277200.748350.457410.23420
Yunnan Univ18.896261.540100.98286.314170.681390.469310.23321
Nankai Univ18.091271.334320.969405.894320.827240.462400.18730
Univ of S&T – China17.782281.303370.968415.797380.850220.47750.17238
Tianjin Univ17.466291.296380.970355.852370.819250.475110.17735
South China Univ of Technol17.286301.349310.970365.882330.795280.473200.18131
Chongqing Univ17.029311.313340.970345.856360.795280.47830.17636
Hunan Univ16.958321.307360.973305.913310.772320.48010.18032
Ocean Univ of China16.824331.73410.977216.102260.677400.473170.20726
Dalian Univ of Technol16.509341.315330.974295.922300.756340.47840.18033
Harbin Inst of Technology15.412351.286400.969385.769390.776310.47580.16539
Beihang Univ15.007361.311350.970375.762400.780300.465380.16340
China Agricultural Univ14.671371.67030.973315.873350.701370.469340.17637
Northwest A&F Univ14.040381.68120.972325.881340.665410.469330.17734
Beijing Inst of Technol13.944391.269440.969395.728410.724360.47560.15941
Xinjiang Univ12.921401.369300.975285.997280.571420.463390.19329
Univ of Electronic S&T of China12.847411.281420.950445.428430.768330.448430.14742
Minzu Univ of China12.104421.464190.977196.049270.535440.448420.19927
Northwestern Polytechnical Univ12.062431.275430.962425.571420.693380.469320.14643
National Univ of Defense Technol7.783441.285410.951435.274440.563430.446440.12244

Spearman's correlations for ranking order generated by Interd.vb.exe (N = 42).

DIV*TRUE RSVARIETYDISPARITY(1 -GINI)SIMPSON SHANNON
DIV*
TRUE RS.563**
VARIETY.926**.323*
DISPARITY.215−.092.230
(1 – GINI).936.717**.772**.074
SIMPSON.789**.766**.551**.085.917**
SHANNON.911**.734**.725**087.990**.950**

The differences among provided programs.

RoutineName of program requiredInputOutputOther outputWebsite
Interd_vb.exeInterdis.exe*.txt file containing comma-separated variablesInterdis.csv Interdis.dbfhttps://www.leydesdorff.net/software/interdisc.2020/
Syn3_vb.exeSyn3.exe*.txt file containing comma-separated variablessynergy.csv synergy.dbfMinus.net; minus.txt; t_edges.dbf, t_nodes.dbf, t123.dbfhttps://www.leydesdorff.net/software/synergy.triads/
Mode2div.exe.net file (Pajek) formatDiv_col.dbfhttps://www.leydesdorff.net/
jcitnetw.exeWoS downloadsCR as input for mode2divhttps://www.leydesdorff.net/software/interdisc/index.htm

Local vs. global disparity using JCR data for 2019.

UniversityDIV*RankTRUE RSRank
Stanford Univ72.95615.4882
Sun Yat-Sen Univ68.42924.7415
Zhejiang Univ63.34334.30012
Peking Univ62.65444.6328
Shanghai Jiao Tong Univ60.90754.6437
Sichuan Univ58.36764.03319
Harvard Univ58.30174.4619
Wuhan Univ56.90384.7026
Northeastern Univ54.88794.16115
Fudan Univ54.196103.89722
Shandong Univ53.921114.16214
East China Normal Univ51.757124.30911
Xiamen Univ51.354133.70524
Tongji Univ51.348144.8234
Beijing Normal Univ50.815155.0823
Huazhong Univ of S&T50.632163.96320
Central South Univ50.535174.10717
Nanjing Univ50.285183.85123
Lanzhou Univ47.622194.08718
Jilin Univ46.049203.29234
Xi’an Jiaotong Univ45.655213.66425
Tsinghua Univ45.121223.60126
Zhengzhou Univ43.389233.44229
Southeast Univ39.662243.90221
Renmin Univ37.896255.5631
Nankai Univ36.427262.95043
Yunnan Univ36.236274.38210
Univ of S & T – China35.002282.87644
Tianjin Univ34.613292.99541
South China Univ of Technol34.388302.97842
Ocean Univ of China32.747314.20213
Chongqing Univ32.519323.26035
Hunan Univ32.394333.37932
Dalian Univ of Technol31.933343.35533
Harbin Inst of Technol30.166353.19136
Beihang Univ30.029363.50828
China Agricultural Univ29.258373.39631
Northwest A & F Univ27.904383.40230
Beijing Inst of Technol27.102393.18437
Univ of Electronic S&T of China26.892403.07339
Xinjiang Univ25.828413.53127
Northwestern Polytechnical Univ23.873423.03140
Minzu Univ of China22.645434.13216
National Univ of Defense Technol16.236443.11838

Spearman's correlations for consistency of rank order – local vs. global disparity.

DIV*_localTRUE RS_localDIV*_globalTRUE RS_global
DIV*_local
TRUE RS_local.502**
DIV*_global.996**.516**
TRUE RS_global.697**.707**.695**

Abramo, G., Ciriaco Andrea D’Angelo, & Zhang, L. (2018). A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list. Journal of Informetrics, 12(4), 1182–1193.AbramoG.D’AngeloCiriaco AndreaZhangL.2018A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference listJournal of Informetrics12411821193Search in Google Scholar

Ahlgren, P., Jarneving, B., & Rousseau, R. (2003). Requirements for a cocitation similarity measure, with special reference to pearson's correlation coefficient. Journal of the American Society for Information Science and Technology, 54(6), 550–560.AhlgrenP.JarnevingB.RousseauR.2003Requirements for a cocitation similarity measure, with special reference to pearson's correlation coefficientJournal of the American Society for Information Science and Technology546550560Search in Google Scholar

Brewer, D.J., Gates, S.M., & Goldman, C.A. (2001). In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education. Piscataway, NJ: Transaction Publishers, Rutgers University.BrewerD.J.GatesS.M.GoldmanC.A.2001In Pursuit of Prestige: Strategy and Competition in U.S. Higher EducationPiscataway, NJTransaction Publishers, Rutgers UniversitySearch in Google Scholar

Buchan, I. (2002). Calculating the Gini coefficient of inequality. https://www.nibhi.org.uk/Training/Statistics/Gini%20coefficient.doc.BuchanI.2002Calculating the Gini coefficient of inequalityhttps://www.nibhi.org.uk/Training/Statistics/Gini%20coefficient.doc.Search in Google Scholar

Carley, S., Porter, A.L., & Leydesdorff, I.R.L. (2017). Visualization of disciplinary profiles: Enhanced science overlay maps. Journal of Data and Information Science, 2(3), 68–111.CarleyS.PorterA.L.LeydesdorffI.R.L.2017Visualization of disciplinary profiles: Enhanced science overlay mapsJournal of Data and Information Science2368111Search in Google Scholar

Dixon, P.M., & Weiner, J. (1987). Mitchell-Olds T, Woodley R. Boot-strapping the Gini coefficient of inequality. Ecology, 68, 1548–1551.DixonP.M.WeinerJ.Mitchell-OldsTWoodleyR1987Boot-strapping the Gini coefficient of inequalityEcology6815481551Search in Google Scholar

Egghe, L., & Leydesdorff, L. (2009). The Relation between Pearson's correlation coefficient r and Salton's cosine measure. Journal of the American Society for Information Science and Technology, 60(5), 1027–1036.EggheL.LeydesdorffL.2009The Relation between Pearson's correlation coefficient r and Salton's cosine measureJournal of the American Society for Information Science and Technology60510271036Search in Google Scholar

Frenken, K., Oort, F. van, & Verburg, T.N. (2007). Related variety, unrelated variety and regional economic growth. Regional Studies, 41(5), 685–697.FrenkenK.OortF. vanVerburgT.N.2007Related variety, unrelated variety and regional economic growthRegional Studies415685697Search in Google Scholar

Griliches, Z. (1994). Productivity, R&D and the Data constraint. American Economic Review, 84(1), 1–23.GrilichesZ.1994Productivity, R&D and the Data constraintAmerican Economic Review841123Search in Google Scholar

Halffman, W., & Leydesdorff, L. (2010). Is inequality among universities increasing? Gini coefficients and the elusive rise of Elite Universities. Minerva, 48(1), 55–72.HalffmanW.LeydesdorffL.2010Is inequality among universities increasing? Gini coefficients and the elusive rise of Elite UniversitiesMinerva4815572Search in Google Scholar

Herfindahl, O.C. (1950). Concentration in the U.S. steel industry. New York: Columbia University.HerfindahlO.C.1950Concentration in the U.S. steel industryNew YorkColumbia UniversitySearch in Google Scholar

Hirschman, A.O. (1945). National power and the structure of foreign trade. Berkeley: University of California Press.HirschmanA.O.1945National power and the structure of foreign tradeBerkeleyUniversity of California PressSearch in Google Scholar

Kessler, M.M. (1963). Bibliographic coupling between scientific papers. American Documentation, 14, 10–25.KesslerM.M.1963Bibliographic coupling between scientific papersAmerican Documentation141025Search in Google Scholar

Leinster, T., & Cobbold, C.A. (2012). Measuring diversity: The importance of species similarity. Ecology, 93(3), 477–489.LeinsterT.CobboldC.A.2012Measuring diversity: The importance of species similarityEcology933477489Search in Google Scholar

Leydesdorff, L. (2006). Can scientific journals be classified in terms of aggregated journal-journal citation relations using the Journal Citation Reports? Journal of the Association for Information Science and Technology, 57(5), 601–613.LeydesdorffL.2006Can scientific journals be classified in terms of aggregated journal-journal citation relations using the Journal Citation Reports?Journal of the Association for Information Science and Technology575601613Search in Google Scholar

Leydesdorff, L. (2015). Can technology life-cycles be indicated by diversity in patent classifications? The crucial role of variety. Scientometrics, 105(3), 1441–1451.LeydesdorffL.2015Can technology life-cycles be indicated by diversity in patent classifications? The crucial role of varietyScientometrics105314411451Search in Google Scholar

Leydesdorff, L., & Ivanova, I. (2021). The Measurement of “Interdisciplinarity” and “Synergy” in Scientific and Extra-Scientific Collaborations. Journal of the Association for Information Science and Technology, 72(1), 387–402. doi: https://doi.org/10.1002/asi.24416LeydesdorffL.IvanovaI.2021The Measurement of “Interdisciplinarity” and “Synergy” in Scientific and Extra-Scientific CollaborationsJournal of the Association for Information Science and Technology721387402doi: https://doi.org/10.1002/asi.24416Search in Google Scholar

Leydesdorff, L., & Rafols, I. (2011). Indicators of the interdisciplinarity of journals: Diversity, centrality, and citations. Journal of Informetrics, 5(1), 87–100.LeydesdorffL.RafolsI.2011Indicators of the interdisciplinarity of journals: Diversity, centrality, and citationsJournal of Informetrics5187100Search in Google Scholar

Le ydesdorff, L., & Schank, T. (2008). Dynamic animations of journal maps: Indicators of structural changes and interdisciplinary developments. Journal of the American Society for Information Science and Technology, 59(11), 1810–1818.Le ydesdorffL.SchankT.2008Dynamic animations of journal maps: Indicators of structural changes and interdisciplinary developmentsJournal of the American Society for Information Science and Technology591118101818Search in Google Scholar

Leydesdorff, L., Wagner, C.S., & Bornmann, L. (2018). Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene Garfield. Scientometrics, 114(2), 567–592.LeydesdorffL.WagnerC.S.BornmannL.2018Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene GarfieldScientometrics1142567592Search in Google Scholar

Leydesdorff, L., Wagner, C.S., & Bornmann, L. (2019). Diversity measurement: Steps towards the measurement of interdisciplinarity? Journal of Informetrics, 13(3), 904–905.LeydesdorffL.WagnerC.S.BornmannL.2019Diversity measurement: Steps towards the measurement of interdisciplinarity?Journal of Informetrics133904905Search in Google Scholar

Leydesdorff, L., Wagner, C.S., & Bornmann, L. (2019). Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling Diversity, Relative Variety, and the Gini coefficient. Journal of Informetrics, 13(1), 255–264.LeydesdorffL.WagnerC.S.BornmannL.2019Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling Diversity, Relative Variety, and the Gini coefficientJournal of Informetrics131255264Search in Google Scholar

Leydesdorff, L. & Zhou, P. (2007). Nanotechnology as a field of science: Its delineation in terms of journals and patents. Scientometrics, 70(3), 693–713.LeydesdorffL.ZhouP.2007Nanotechnology as a field of science: Its delineation in terms of journals and patentsScientometrics703693713Search in Google Scholar

Leydesdorff, L., Wagner, C.S., & Zhang, L. (2021). Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA. Journal of Digital and Information Science and Technology JDIST; arXiv preprint arXiv:2011.08591.LeydesdorffL.WagnerC.S.ZhangL.2021Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USAJournal of Digital and Information Science and Technology JDISTarXiv preprint arXiv:2011.08591.Search in Google Scholar

Liu, X. (2018). The ‘double first class’ initiative under top-level design. ECNU review of education. 1(1), 147–152.LiuX.2018The ‘double first class’ initiative under top-level designECNU review of education11147152Search in Google Scholar

Lorenz, M. (1905). Methods of Measuring the Concentration of Wealth. Publications of the American Statistical Association, 9(70), 209–219.LorenzM.1905Methods of Measuring the Concentration of WealthPublications of the American Statistical Association970209219Search in Google Scholar

Mills, J.A., & Zandvakili, A. (1997). Statistical inference via bootstrapping for measures of inequality. Journal of Applied Econometrics, 12, 133–150.MillsJ.A.ZandvakiliA.1997Statistical inference via bootstrapping for measures of inequalityJournal of Applied Econometrics12133150Search in Google Scholar

Nijssen, D., Rousseau, R., & Hecke, P.V. (1998). The Lorenz curve: A graphical representation of evenness. 13(1), 33–38.NijssenD.RousseauR.HeckeP.V.1998The Lorenz curve: A graphical representation of evenness1313338Search in Google Scholar

Pfeffer, J., Mrvar, A., & Batagelj, V. (2013). txt2pajek: Creating Pajek Files from Text Files. Technical Report, 110, CMU-ISR-13.PfefferJ.MrvarA.BatageljV.2013txt2pajek: Creating Pajek Files from Text FilesTechnical Report, 110, CMU-ISR-13.Search in Google Scholar

Porter, A.L., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719–745.PorterA.L.RafolsI.2009Is science becoming more interdisciplinary? Measuring and mapping six research fields over timeScientometrics813719745Search in Google Scholar

Powell, W.W., & DiMaggio, P.J. (Eds.). (2012). The new institutionalism in organizational analysis. Chicago: University of Chicago Press.PowellW.W.DiMaggioP.J.(Eds.).2012The new institutionalism in organizational analysisChicagoUniversity of Chicago PressSearch in Google Scholar

Pratt, A.D. (1977). A measure of class concentration in bibliometrics. Journal of the American Society for Information Science, 28(5), 285–292.PrattA.D.1977A measure of class concentration in bibliometricsJournal of the American Society for Information Science285285292Search in Google Scholar

Rafols, I. (2014). Knowledge integration and diffusion: Measures and mapping of diversity and coherence. In Measuring Scholarly Impact (pp. 169–190). Springer, Cham.RafolsI.2014Knowledge integration and diffusion: Measures and mapping of diversity and coherenceInMeasuring Scholarly Impact169190SpringerChamSearch in Google Scholar

Rafols, I., & Meyer, M. (2007). How cross-disciplinary is bionanotechnology? Explorations in the specialty of molecular motors. Scientometrics, 70(3), 633–650.RafolsI.MeyerM.2007How cross-disciplinary is bionanotechnology? Explorations in the specialty of molecular motorsScientometrics703633650Search in Google Scholar

Rafols, I., & Meyer, M. (2010). Diversity and network coherence as indicators of interdisciplinarity: Case studies in bionanoscience. Scientometrics, 82(2), 263–287.RafolsI.MeyerM.2010Diversity and network coherence as indicators of interdisciplinarity: Case studies in bionanoscienceScientometrics822263287Search in Google Scholar

Rafols, I., Porter, A.L., & Leydesdorff, L. (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology, 61(9), 1871–1887.RafolsI.PorterA.L.LeydesdorffL.2010Science overlay maps: A new tool for research policy and library managementJournal of the American Society for Information Science and Technology61918711887Search in Google Scholar

Rousseau, R. (1992). Concentration and diversity measures in informetric research. Ph.D. Thesis, University of Antwerp.RousseauR.1992Concentration and diversity measures in informetric researchPh.D. Thesis,University of AntwerpSearch in Google Scholar

Rousseau, R. (2018). The repeat rate: From Hirschman to Stirling, Scientometrics, 116(1), 645–653.RousseauR.2018The repeat rate: From Hirschman to StirlingScientometrics1161645653Search in Google Scholar

Rousseau, R. (2019). On the Leydesdorff-Wagner-Bornmann proposal for diversity measurement. Journal of Informetrics, 13(3), 906–907.RousseauR.2019On the Leydesdorff-Wagner-Bornmann proposal for diversity measurementJournal of Informetrics133906907Search in Google Scholar

Salton, G., & McGill, M.J. (1983). Introduction to Modern Information Retrieval. Auckland, etc: McGraw-Hill.SaltonG.McGillM.J.1983Introduction to Modern Information RetrievalAuckland, etcMcGraw-HillSearch in Google Scholar

Shannon, C.E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423.ShannonC.E.1948A mathematical theory of communicationThe Bell System Technical Journal273379423Search in Google Scholar

Simpson, E.H. (1949). Measurement of diversity. Nature, 163(4148), 688–688.SimpsonE.H.1949Measurement of diversityNature1634148688688Search in Google Scholar

Stirling, A. (1998). On the economics and analysis of diversity. Science Policy Research Unit (SPRU), Electronic Working Papers Series, Paper, 28, 1–156.StirlingA.1998On the economics and analysis of diversityScience Policy Research Unit (SPRU)Electronic Working Papers Series, Paper, 281156Search in Google Scholar

Stirling, A. (2007). A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface, 4(15), 707–719.StirlingA.2007A general framework for analysing diversity in science, technology and societyJournal of the Royal Society Interface415707719Search in Google Scholar

Wagner, C.S., Bornmann, L., Cai, X., & Leydesdorff, L. (in preparation). Isomorphism as an Ordering Dynamic in the Growth of China's Science System.WagnerC.S.BornmannL.CaiX.LeydesdorffL.(in preparation).Isomorphism as an Ordering Dynamic in the Growth of China's Science SystemSearch in Google Scholar

Waltman, L., Eck, N.J., & Noyons, E.C.M. (2010). A unified approach to mapping and clustering of bibliometric networks. Journal of Informetrics, 4(4), 629–635.WaltmanL.EckN.J.NoyonsE.C.M.2010A unified approach to mapping and clustering of bibliometric networksJournal of Informetrics44629635Search in Google Scholar

Whitley, R.D. (1984). The Intellectual and Social Organization of the Sciences. Oxford: Oxford University Press.WhitleyR.D.1984The Intellectual and Social Organization of the SciencesOxfordOxford University PressSearch in Google Scholar

Zhang, L., Rousseau, R., & Glänzel, W. (2016). Diversity of references as an indicator of the interdisciplinarity of journals: Taking similarity between subject fields into account. Journal of the Association for Information Science and Technology, 67(5), 1257–1265.ZhangL.RousseauR.GlänzelW.2016Diversity of references as an indicator of the interdisciplinarity of journals: Taking similarity between subject fields into accountJournal of the Association for Information Science and Technology67512571265Search in Google Scholar

Zhang, L., Sun, B., Chinchilla-Rodríguez, Z., Chen, L., & Huang, Y. (2018). Interdisciplinarity and collaboration: On the relationship between disciplinary diversity in departmental affiliations and reference lists. Scientometrics, 117(1), 271–291.ZhangL.SunB.Chinchilla-RodríguezZ.ChenL.HuangY.2018Interdisciplinarity and collaboration: On the relationship between disciplinary diversity in departmental affiliations and reference listsScientometrics1171271291Search in Google Scholar

Zhang, L., Sun, B., Jiang, L., & Huang, Y. (2021). On the relationship between interdisciplinarity and impact: Distinct effects on academic and broader impact, Research Evaluation, rvab007. https://doi.org/10.1093/reseval/rvab007ZhangL.SunB.JiangL.HuangY.2021On the relationship between interdisciplinarity and impact: Distinct effects on academic and broader impactResearch Evaluationrvab007. https://doi.org/10.1093/reseval/rvab007Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo