Beyond surface correlations: Reference behavior mediates the disruptiveness-citation relationship

We begin by employing the basic Poisson regression model to examine the association between the CD index and citation count without introducing any controls. Specifically, we conducted split-sample regressions based on the publication year of the papers from 1950 to 2016. This approach provides initial insights into the evolving relationship between the CD index and citation impact over time.

The findings illustrated in Figure 2 reveal a noteworthy pattern. Prior to 1962, there was a statistically significant positive association between the CD index and citation count, indicating that papers with a higher CD index experienced a higher citation impact during this period. However, subsequent to this time point, the coefficient turns negative, with the exception of the years 1974 and 1975, where the coefficient is statistically insignificant. This trend persists until the present, suggesting that since then, papers with a higher CD index have been linked to a lower citation impact. Notably, the coefficient exhibits a declining trajectory (except for the latest two years, which may be due to the incomplete data in its 5-year citation windows), indicating that over time, papers with a high CD index are increasingly associated with diminished citation impact.

Figure 2.

Basic yearly split-sample Poisson regression coefficients of the CD index on citation counts from 1950 to 2016. Error bars depict the upper and lower bounds of the 95% confidence intervals based on robust standard errors. Dark-red coloring indicates significant positive coefficients (p < 0.05), and light-red coloring indicates non-significant positive coefficients. Dark-green coloring indicates significant negative coefficients (p < 0.05), light-green coloring indicates non-significant negative coefficients. * p < 0.05, ** p < 0.01, *** p < 0.001.

This result aligns with the findings of Zeng et al. (2023) and remains robust across various models (Appendix A), irrespective of whether 5-year windows are employed (Appendix A). Recognizing the skewed distribution of the CD index (Yang, Deng, et al., 2023), we transformed the index into percentiles, calculated within each publication year and field (using the 19 level-0 fields of MAG). This normalization process did not change the overall conclusions (Appendix A).

It is important to note that Funk and Owen-Smith (2017) contended that the CD index was designed to capture a concept distinct from citation impact. They also introduced the mCD index, which is the product of the CD index and citation count. Although the mCD index demonstrated a consistent positive relationship with citation count, the coefficients exhibited a declining trend over time (Appendix A). This further reinforces our argument regarding the increasing bias against the CD index.

In this subsection, we augment the baseline Poisson regression models with fixed effects to elucidate the causes behind the negative association observed between the CD index and citation impact. The rationale for employing fixed effects instead of control variables is their capacity to address heterogeneity more comprehensively, thereby facilitating a deeper understanding of the underlying mechanisms.

Initially, we hypothesized that the observed negative association between the CD index and citation count may stem from heterogeneity across different fields of study. Variations in citation norms among distinct fields, such as the propensity for Medicine and Biology papers to garner more citations compared to social science papers of similar quality (Radicchi et al., 2008), could contribute to this phenomenon. Additionally, prior research by Chu and Evans (2021) indicated that larger fields tend to exhibit lower average CD index values. To explore this hypothesis, we introduced fixed effects representing 292 second-level fields into the regression model. The results depicted in Figure 3a suggest that the heterogeneity across different fields of study does not suffice to explain the escalating bias against the CD index.

Figure 3.

Yearly split-sample Poisson regression coefficients of the CD index on citation counts from 1950 to 2016, with (a) 292 fields fixed effect, (b) reference count fixed effect, (c) team size fixed effects, and (d) all fixed effects. Error bars depict the upper and lower bounds of the 95% confidence intervals based on robust standard errors. Dark-red coloring indicates significant positive coefficients (p < 0.05), and light-red coloring indicates non-significant positive coefficients. Dark-green coloring indicates significant negative coefficients (p < 0.05), light-green coloring indicates non-significant negative coefficients. * p < 0.05, ** p < 0.01, ***p < 0.001.

Subsequently, we considered the potential influence of team size on the observed negative association between the CD index and citation count. The increasing prevalence of large teams in scientific endeavors may significantly alter research patterns (Jones et al., 2008; Wuchty et al., 2007). Notably, large teams often accrue higher citation counts than papers authored by individuals. Moreover, recent studies (Lin et al., 2023; Wu et al., 2019) propose that small, locally based teams may foster more disruptive research outcomes. To investigate this hypothesis, we introduced fixed effects representing team size (the number of authors) into the regression model. However, as indicated in Figure 3c, the heterogeneity in the different types of teams does not adequately explain the escalating bias against the CD index.

Finally, we propose that referencing behavior may contribute to the observed negative correlation between the CD index and citation count. This can be attributed to the sensitivity of the CD index to the number of references within the papers (Leibel & Bornmann, 2024). For instance, a paper with fewer references could potentially generate a high CD index because future citations are less likely to reference its limited sources. Conversely, papers with a higher number of references tend to have higher citation counts (Zeng et al., 2017). To test this hypothesis, we integrated reference count fixed effects into the regression model, with the maximum set at 60 (with counts above 60 capped at 60). As depicted in Figure 3b, upon inclusion of the reference count fixed effect, the coefficients exhibit positive significance, consistently observed from 1950 to 2016, without displaying a declining trend over time. The full model, encompassing all fixed effects, yields a similar outcome to the model solely incorporating reference fixed effects, yet significantly diverges from the other models (Figure 3d). Thus, we assert that referencing behavior can substantially elucidate escalating bias against the CD index.

To further validate our findings, we conducted an additional set of split-sample Poisson regression analyses utilizing papers’ reference counts as the split rather than their publication years. This approach aimed to ascertain whether the observed negative association between the CD index and citation count could be explained by variations in referencing behavior. Our hypothesis is that if reference count adequately captures the influence of the CD index on citation count, then the coefficients of the CD index in split-sample Poisson regressions, stratified by reference count, should consistently exhibit positive effects, assuming that disruptive papers are more impactful. Essentially, by analyzing papers with equivalent reference counts, we aimed to isolate the impact of the CD index on citation counts from referencing behavior.

The results depicted in Figure 4 align with our hypotheses. When examining the relationship between the CD index and citation count within papers sharing the same reference count, ranging from 1 to 59 and above 60, all subsamples demonstrated positive effects, with the majority being statistically significant. Furthermore, upon incorporating additional fixed effects into the models, including fields, team level, author level, and year fixed effects, all subsamples exhibited significantly positive results at the 0.001 significance level. Consequently, we assert that referencing behavior substantially contributes to the elucidation of the observed bias against the CD index. However, this analysis underscores a limitation: by strictly controlling for reference count in our fixed effects regression, we may obscure the reduced citation impact of highly disruptive papers that cite fewer references, suggesting that our method may not fully capture the dynamics of disruption-driven citation patterns.

Figure 4.

Split-sample Poisson regression coefficients of the CD index on citation counts based on reference counts. (a) Basic models without any fixed effects. (b) Full models including all fixed effects. Error bars depict the upper and lower bounds of the 95% confidence intervals based on robust standard errors. * p < 0.05, ** p < 0.01, *** p < 0.001.

In this subsection, we analyze additional potential channels that could explain the negative relationship between the CD index and citation count. To ensure that the results are intuitive, we focused on a subset of the dataset, considering only papers published between 1990 and 2010. This period was chosen because the negative relationship between the CD index and citation count was most significant during these years, even when controlling for the field (see Figure 3a). We utilized Poisson regression models, incorporating both year- and field-fixed effects in all analyses to obtain unbiased results.

First, we consider the potential influence of team dynamics on the observed negative association between the CD index and citation count. While in Figure 3c we only analyzed the control of team size, many other factors could also influence the disruptive potential of papers, such as cross-institutional collaboration (Yang, 2024, 2025), international collaboration (Lin et al., 2023), interdisciplinary collaboration (Liu et al., 2024), and gender diversity (Yang et al., 2022). Detailed definitions of these variables are provided in the Appendix C.

The results in Table 1 reveal that none of the team-level control variables can account for the negative relationship between the CD index and citation count. The coefficient of the CD index on citation count remains negatively significant in Models (1–7) after we control for these team-level variables.

Second, we hypothesize that the experience and academic status of authors may contribute to the observed negative association between the CD index and citation count. It is plausible that advancing age and increased academic stature may lead authors to produce less disruptive research due to the burden of knowledge (Bloom et al., 2020; Jones, 2009; Jones & Weinberg, 2011; Li, Tessone, et al., 2024), while also potentially enhancing the citation counts of their papers (Yang, Xu, et al., 2024). To test this hypothesis, we introduced a set of author career-level variables, including the average career age of authors, average and maximum career productivity of authors, and average and maximum career citations of authors. These variables are dynamic and are calculated only up to the publication year of the focal paper. Detailed definitions of these variables are provided in Appendix C.

As shown in Table 2, author career age cannot fully account for the observed negative association between the CD index and citation count. The coefficient in Model (2) remained negatively significant, although the negative effect size was much smaller. However, both author career productivity and citations can account for this negative association. The coefficients in Models (3-6) are all positively significant, and the effect size is larger when controlling for author career citations and maximum author features. When combining all author career-level controls in Model (7), the coefficient of the CD index is positively significant and exhibits a larger effect size.

While these results suggest that author career level can partially explain the observed negative association between the CD index and citation count, it remains unclear whether this is due to differences in reference behavior or other factors. A deeper exploration of the mechanisms at the author level is beyond the scope of this study.

Finally, we further investigated additional reference-level factors that may influence the association between the CD index and citation count. The literature indicates that reference count, reference age, and citations of references (i.e. reference popularity) can all affect the CD index (Leibel & Bornmann, 2024; Li, Lin, et al., 2024). Unlike author-level and team-level factors, these reference-level influences may stem from the definition and calculation of the CD index itself.

The results presented in Table 3 reveal that the reference count significantly accounts for the observed negative association between the CD index and citation count. When controlling for reference count or reference age, the effect of the CD index on citation count transitions from negatively significant to positively significant. When accounting for the citations of references, the coefficient transitions from being negatively significant to insignificant at the 0.05 level. When combining all reference-level controls in Model (6), the coefficient increases to 1.92, which is significantly larger than the original estimate.

Although these coefficients should be interpreted with caution, the model suggests notable changes in the association. In Model (1), without controlling for reference-level variables, an increase of one standard deviation in the CD index (0.14, see Appendix C) is associated with a 2.8% decrease in citations (exp(-0.20*0.14) - 1). However, after accounting for all reference-level variables in Model (6), an increase of one standard deviation in the CD index is associated with a 30.1% increase in citations (exp(1.92*0.14) - 1).

Based on these findings, we assert that the observed increasing bias against the CD index (a negative association between the CD index and citation count) can be attributed to changing referencing behaviors in scholarly papers over time. As depicted in Figure 5a, the average number of references in papers has steadily increased over the past seven decades, from 10 to 37. This upward trend in referencing behavior suggests a temporal shift in scholarly reference practices. The average 5-year citation count has also increased, climbing from 5 to 19 over the same period (Figure 5b), and the average 5-year CD index has declined from 0.09 to 0.03 (Figure 5c). Our earlier findings suggest that when controlling for the reference count, the association between the CD index and citation count remains consistently positive. Therefore, we redirect our focus to examining the relationship between the variables of interest and reference count.

Figure 5.

Reference counts as a contributing factor to the bias against CD index. (a) The average reference count of papers increases over time. (b) The average 5-year citation count of papers increases over time. (c) The average 5-year CD index of papers decreases over time. (d) Papers with higher reference count are associated with higher 5-year citation count. (e) Papers with higher reference count are associated with higher 5-year CD index. (f) Papers with higher reference count are associated with higher parameters of 5-year CD index (ni, nj, and nk). (g) The number of papers with different reference count follows lognormal distribution. (h-i) The complementary cumulative distribution function (ccdf) and probability density function (pdf) of the CD index for papers with varying reference counts. Shaded areas indicate 95% confidence intervals.

The results depicted in Figure 5d demonstrate a linear association between higher reference counts and higher 5-year citation counts. Conversely, the findings in Figure 5e indicate that a higher reference count is linearly associated with a lower 5-year CD index. Thus, as the average number of references in papers increases over time, the prevalence of papers with a high number of references contributes to the negative association between the CD index and citation count.

Finally, we propose a potential mechanism underlying the influence of reference behavior on this association, elucidated through the parameters of the 5-year CD index (n_i, n_j, and n_k). The findings depicted in Figure 5f suggest that the impact of the reference count on distinct parameters varies. Considering n_i as indicative of disruptive citations and n_j as representative of consolidating citations, the analysis revealed that a higher reference count corresponds to a greater increase in n_j, subsequently resulting in a decrease in the proportion of n_i. This, in turn, leads to a reduction in the CD index. Furthermore, a higher value of n_k also contributes to a lower CD index. In our dataset, where the average CD index exceeded 0, a larger denominator resulted in a diminished CD index.

The mechanism described can substantially impact the outcomes of analyses involving the CD index. As depicted in Figures 5h–i, papers with a limited number of references are likely to display extreme values on the CD index—either significantly high or low—while those with a larger number of references tend to show CD index values clustered around zero. This mechanism underscores the sensitivity of the CD index to reference count, which, combined with our fixed effects regression method’s strict control for reference count, may obscure the citation dynamics of highly disruptive papers with fewer references. Consequently, while reference behavior explains the methodological bias in the CD index, our analytical approach may not fully resolve whether disruptive papers are losing impact, suggesting the need for alternative regression methods or metrics to capture this phenomenon.

Following the methodology of Arts et al. (2025), we processed and integrated their datasets to identify new words and their reuse. Specifically, we analyzed English, non-retracted journal publications from OpenAlex, leveraging a sophisticated text-processing pipeline. Using the spaCy model “en_core_sci_lg,” the pipeline includes tokenization, POS-tagging, dependency parsing, chunking, lemmatization, cleaning, baseline removal, and vectorization. This approach enables us to track the first appearance of words and their subsequent reuse to quantify innovation. Notably, this method is entirely text-based and independent of citation or reference information.

We begin by examining the relationship between the number of new words (log-transformed) and the 5-year citation count using a basic Poisson regression model, without introducing any controls. By conducting split-sample regressions across publication years (1950–2016), we find that the relationship between new words and citation count remains consistently positive (Figure 6a). This relationship is statistically significant at the 0.001 level for nearly all years, except 1950 and 1951, suggesting a distinct trend from the CD index. Furthermore, we assessed the impact of reference-level factors on this relationship. As shown in Table 4, controlling for reference-level variables did not alter the direction, effect size, or significance of the association. Across all models, we observe a robust, statistically significant (0.001 level) positive relationship between new words (log-transformed) and the 5-year citation count.

Using the same procedure, we then analyze the relationship between new word reuse (log-transformed) and the 5-year citation count. As shown in Figure 6b, this relationship also remained consistently positive and statistically significant at the 0.001 level. Similarly, Table 5 demonstrates that controlling for reference-level variables did not alter the direction, effect size, or significance of the association. Across all models, we find a robust, statistically significant (0.001 level) positive relationship between new word reuse (log-transformed) and the 5-year citation count.

Figure 6.

Yearly split-sample Poisson regression coefficients of new words, new word combinations, and their reuse on citation counts from 1950 to 2016. Error bars depict the upper and lower bounds of the 95% confidence intervals based on robust standard errors. Dark-red coloring indicates significant positive coefficients (p < 0.05), and light-red coloring indicates non-significant positive coefficients. * p < 0.05, ** p < 0.01, *** p < 0.001.

We then analyzed the relationship between new word combinations and their reuse (both log-transformed) and 5-year citation counts. Here, new word combinations are based on all possible pairs between any of the words in a paper. As shown in Figures 2c–d, this relationship also remained consistently positive and statistically significant at the 0.001 level. Similarly, Tables 6–7 demonstrates that controlling for reference-level variables does not alter the direction, effect size, or significance of the association. Across all models, we find a robust, statistically significant (0.001 level) positive relationship between new word combinations and their reuse (both log-transformed) and 5-year citation count.

To measure atypical combinations of knowledge, we adopted the framework of Uzzi et al. (2013), analyzing the distribution of journal pairs within a paper’s reference list. Each journal pairing was converted into z-scores, representing standardized measures of novelty. We quantified atypical combinations using Monte Carlo simulations, which generate reshuffled citation networks by randomly reassigning edges while preserving the temporal and distributional properties of the original network. The atypicality score for each journal pair is calculated as follows: 5Zscorei,j=obs(pairij)−exp(pairij)σ(pairij) where obs(pair_ij) signifies the observed frequency of the journal pair in the real dataset, exp(pair_ij) represents the mean, and σ(pair_ij) denotes the standard deviation of the journal pairs obtained from 10 randomized simulations of the reshuffled network.

To summarize the distribution of z-scores, we extracted the 10th percentile value for each reference list. Given that the distribution of these values is highly skewed, we normalized atypicality scores into percentiles within each publication year and field (across 19 Level-0 fields in MAG).

Next, we analyze the relationship between atypicality (percentile) and the 5-year citation count using a basic Poisson regression model without additional controls. Split-sample regressions across publication years (1950–2016) revealed a consistently positive relationship between atypicality and citation count (Figure 7a). This association remained statistically significant at the 0.001 level for nearly all years, diverging from the trend observed for the CD index. Furthermore, as shown in Table 8, incorporating reference-level controls slightly reduces the effect size (especially the number of references) but does not alter the direction or significance of the association. Across all models, we find a robust, highly significant (0.001 level) positive relationship between atypicality and the 5-year citation count.

Figure 7.

Yearly split-sample Poisson regression coefficients of atypicality and disruptive citations on citation count from 1950 to 2016. Error bars depict the upper and lower bounds of the 95% confidence intervals based on robust standard errors. Dark-red coloring indicates significant positive coefficients (p < 0.05), and light-red coloring indicates non-significant positive coefficients. * p < 0.05, ** p < 0.01, *** p < 0.001.

Beyond the CD index, which captures the relative proportion of disruptive impact, we employ disruptive citations to measure the absolute disruptive impact of papers, defined as: 6Disruptive citation=ni

Disruptive citations follow a power-law distribution (Yang et al., 2025), similar to other network-based metrics, providing a more granular perspective on a paper’s disruptive impact. Unlike the CD index, this metric is independent of the focal paper’s reference list, thereby eliminating potential biases introduced by the citing behavior of the paper itself (Yang & Deng, 2024; Yang, Hu, et al., 2023; Yang et al., 2025). Additionally, its broad range enables more precise differentiation between papers with varying levels of disruptive impact, making it a powerful tool for ranking scholarly contributions. Prior research has shown that disruptive citations effectively identify milestone papers and laureates (Lin et al., 2022; Yang, Hu, et al., 2023) while sheddinglight on the relationship between scientific impact and disruption.

We examined the relationship between disruptive citations and the 5-year citation count using a Poisson regression model without additional controls. Our split-sample regressions across publication years (1950–2016) revealed a consistently positive relationship between disruptive citations and citation count (Figure 7b), with statistical significance at the 0.001 level for nearly all years. Moreover, as Table 9 demonstrates, incorporating reference-level controls did not alter the direction, effect size, or significance of the association. Across all models, we find a robust, highly significant (0.001 level) positive relationship between disruptive citations and the 5-year citation count.