Tracking direct and indirect impact on technology and policy of transformative research via ego citation network

The definition and measurement of societal impact pose persistent challenges, often referring to demonstrable research contributions to society (ESRC, 2022). Recognized as a complex process (Morton, 2015), demonstrating societal impact is deemed essential (Bornmann et al., 2016). Various payback frameworks have emerged (Martin, 2011). Citation-based approaches, explored in health research (Lewison, 2004; Yin et al., 2021), date back to Narin et al.’s work in the 1980s, revealing knowledge flows from public research to patents, especially in biomedicine (McMillan et al., 2000). Li et al. (2017) support arguments on practical applications, finding NIH grants contributing significantly to patents.

Ke’s recent studies in biomedical science and technology indicate exponential growth in patent-to-paper citations (Ke, 2020). Papers citing diverse fields receive more patent citations (Ke, 2023). Gene engineering research influences policy decisions (Bartkowski et al., 2018), and Belardo et al. (2018) illustrated the impact of biomedical knowledge on health policies.

Astrophysical research, akin to Pasteur’s Quadrant, emphasizes the interaction between basic and applied research, fostering impact diffusion. Venturini et al. (2014) summarized technology transfer in astrophysics, and Petroni et al. (2023) concluded that it generates scientific and technological innovations. Thelwall (2016) explored public interest in astrophysics.

Citation-based methods for assessing societal impacts are gaining attention, but they often focus on direct citations. Researchers advocate considering multiple citation generations. Rousseau (1987) is among the earliest to explore both direct and indirect citations. Fragkiadaki et al. (2014) defined citation generations, and Atallah et al. (2006) empirically developed a cumulative citations model. Fragkiadaki et al. (2011) introduced the f-value, incorporating direct and indirect citations. Hu et al. (2012) analyzed patent ego citation networks, proposing indicators based on multiple generations. Recently, Hu et al. (2016) emphasized the significance of indirect citations for research impact. Jones and Hanney (2016) examined five citation generations for four papers, focusing on identifying important citations rather than societal impact assessment.

Various citation generation types can be defined in different contexts. For our analysis, we account for potentially overlapping citations. Generations are viewed as multisets, where elements may appear multiple times across different citation generations (Rousseau et al., 2018). Specifically, we designate a NP publication as generation zero (the target article). Publications citing the target article make up the first-generation citations, and publications citing first-generation citations are considered as second-generation citations (Hu et al., 2011).

In this work, we only consider key Nobel Prize publications in gene engineering and astrophysics as a proxy for TRs. Data collecting processes consist of four steps:

Step 1. Identifying Nobel Prizes in gene engineering and astrophysics: Nobel Prizes in Physiology or Medicine (1978, 2007), Chemistry (2020) for gene engineering; Physics (1936, 1974, 1978, 1983, 1993, 2002, 2006, 2011, 2019, 2020) for astrophysics.

Following the above steps, we collected more than 2,000,000 records in total, including more than 1,300,000 publications, 800,000 patents, and 15,000 policies.

We consider an article citation network and focus on one target article the ego, as it is called in network theory (Rousseau, 2011). Based on ego citation network of each NP (as shown in Figure 1), we investigate its (in)direct impact on technology and policy.

Figure 1.

3.3.1

A network-structural indicator for technological impact

The number of patent citations received by research is not equivalent to its technological impact. Many scholars think that being cited by patents is only an intermediate step to achieving technological impact of an article, rather than the final step (Kim et al., 2017). Instead of counting the number of patent citations, we use the average value of citing patents to assess technological impact of an article.

The value of a patent is complex so that simple citation counting has limitations for comparing potential values of patents across fields. Moreover, neither direct relation with the technological background of a patent nor technological complexity included in a patent could be reflected by citation counting. In our previous article, we introduced the technological span index (TSI), a structural indicator of patent citation network (Hu et al., 2012), to measure the potential value of patents for further innovative development. The TSI takes the subnetwork CR (citation of reference) of a focused patent into account, namely, the references of patent and the number of citations received by those reference items from dynamic networks (as shown in Figure 2). The previous study suggested that the TSI could validly indicate the potential technological value of patents for further innovative development. In this work, we use the average TSI value of all citing patents to assess technological impact of an article in this work. It is calculated by, (1) TSI¯=∑i=1nTSIin \[\overline{TSI}=\frac{\sum\limits_{i=1}^{n}{TS{{I}_{i}}}}{n}\] where n is the number of patents citing the target article. The TSI is based on, but different from patent citations. Taking patent p as an example (as shown in Figure 2), its TSI is calculated based on the CR (citations of references) subnetwork, which is the subnetwork comprising p’s referenced patents and patents citing these reference items. TSI is calculated by, (2) TSI=OIP∗TII \[TSI=O{{I}_{P}}*TII\] where OI_p means the outgrow index of p and TII denotes the technical interest index of p. OI_p reflects the relative position of p in the network comprising p and its reference patents. It is defined as, (3) OIP=1−RPT(p)+1 \[O{{I}_{P}}=1-\frac{{{R}_{P}}}{T\left( p \right)+1}\] where T(p) denotes the number of patents cited by p and R_P denotes the rank of p in the ranked list of T(p)+1 elements in decreasing order by the number of patent citations received (Hu et al., 2012).

TII measures the innovative density of technological knowledge flows reflected by patents cited by p. It is defined as the squared root of the total number of citations of all cited patents (see Figure 2). Mathematically, it is, (4) TII=CITT(RP) \[TII=\sqrt{CI{{T}_{T\left( RP \right)}}}\] where CIT_T(RP) denotes the total number of citations of all cited patents.

We computed the technological impact of more than 1,300,000 publications with Python, and the routine is provided at https://github.com/lx2009xian/TSI_calculation.

Figure 2.

To investigate the differences between NPs and their two generations of citations in technological and policy impact, we created two dummy variables to represent three groups, controlling for potential variables to eliminate interference on impact.

For technological impact analysis, we applied a linear regression model as the dependent variable is continuous. The following variables were included as controls:

Publication age: The year interval from publication to 2022. Older articles typically have a longer citation time window, leading to more citations (Wang, 2013).

N-publication citations: The number of citations a publication received. Highly cited publications tend to have a higher technological impact (Veugelers et al., 2019).

N-categories: The number of research categories a publication belongs to (Zhang et al., 2022). More categories often result in higher technological impact (Campbell et al., 2017).

Time lag: The time gap between publication and the first patent citation. A shorter time lag is associated with higher technological impact (Ke, 2020).

N-references: The number of references in an article. More references often lead to higher impact (Zhang et al., 2021) and longer reference lists are linked to higher technological impact (Veugelers et al., 2019).

For policy impact analysis, we used a negative binomial regression model to account for the over-dispersed nature of policy citations. We controlled for the following variables:

Publication age: The year interval from publication to 2022 (Wang, 2013).

Publication-type: The type of a publication. Different publication types may have varying numbers of policy citations (Bornmann et al., 2016).

N-research countries: The number of countries collaborating on a publication. International collaborations often result in higher impact (Didegah et al., 2018).

N-publication citations: The number of citations a publication received, as it’s positively correlated with policy citations (Heydari et al., 2019).

N-references: The number of references. More references in an article are associated with higher policy citations (Huang et al., 2022).

Table 1 summarizes descriptive statistics for NPs (scientific publications) and their citation generations. In gene engineering, around 64% of NPs receive patent citations, while this probability drops to 14% for the first generation and 10% for the second generation of citations. In astrophysics, these rates are notably lower at 2%, 0.2%, and 0.4%, respectively.

Among scientific publications cited by patents, NPs in gene engineering receive more patent citations and are cited across a broader range of technological fields compared to their citation generations. Conversely, in astrophysics, the trend is reversed. From Table 1, we also investigate that NPs in both fields tend to receive their first patent citation later than their citing articles.

Regarding policy citations, in gene engineering, 14.86% of NPs are cited by policies, compared to 1.68% for the first generation and 1.05% for the second generation of citations. In astrophysics, these percentages are lower, with values of 2.03%, 0.10%, and 0.18%, respectively.

Among scientific publications cited by policies, NPs in gene engineering receive an average of 2.27 policy citations, surpassing the average for their citing articles. In astrophysics, the average policy citations for NPs are lower than that of their citation generations.

In this section, we investigated the impact of NPs and their generations of citations on technology. Figure 3 shows the technological impact of NPs and two generations of citations. Figure 4 shows their differences in technological impact where each dot represents the mean value of technological impact and the vertical line indicates the mean ± standard deviation. Calculated at the mean level, the technological impact for NPs in gene engineering is 9.37, 4.51 for first generation of citation and 3.49 for second generation of citation.

Figure 3.

Figure 4.

We conducted a series of analyses while controlling for potential confounding factors, as previously described. To ensure the accuracy of model estimates and assess the risk of multicollinearity among variables, we performed variance inflation factor (VIF) tests. The results, presented in Table 2, indicate that all variables had VIFs lower than 10, suggesting no significant issues with multicollinearity

We conducted linear regression models with the technological impact as the dependent variable and two dummy variables as the independent variables. The results are displayed in Table 3. Table 3 and Figure 4a reveal that NPs in gene engineering have a significantly higher technological impact compared to their first generation of citations, and the first generation has a significantly higher technological impact than the second generation. These findings are robust, as they remain consistent after controlling for potential confounding variables (Table 3, columns 2-7).

Table 3 also shows that the technological impact is positively correlated with N-publication citations and Publication age, while a negative correlation trend is observed with N-references. Furthermore, comparing column 1 and 2 suggests that additionally controlling for N-publication citations reduces the gap between NPs and their citation generations. N-publication therefore mediates the intergroup difference in technological impact, but it does so only partly. The results in Table 3 indicate that Time lag, N-references and Publication age also play the role in mediating the intergroup difference.

Similar regression analysis was performed for NPs in astrophysics and their citing publications. In this case, the differences in technological impact between NPs in astrophysics and their citation generations are not statistically significant (Figure 4b and Table 3). Even after controlling for several variables (N-publication citations, N-categories, Time lag, N-references, and Publication age), NPs in astrophysics still do not exhibit significantly higher technological impact than subsequent citation generations. It’s worth noting that in astrophysics, technological impact positively correlates with Publication age, indicating earlier publications having a longer citation window and higher technological impact (Ke, 2020).

Table 4 outlines descriptive statistics for policy documents. In gene engineering, policy documents citing NPs originate from six research institutions across four countries, predominantly the U.S. National Academies Press, InterAcademy Partnership in Italy, and the World Health Organization. Further details on policy citations for first-generation and second-generation citations in gene engineering are also provided. For first-generation citations, the top three countries with the highest publication volume of citing policy documents are the United States, Switzerland, and Italy, with the National Academies Press, the World Health Organization, and the International Union for Conservation of Nature being the top three institutions. The corresponding countries for second-generation citations are the United States, Switzerland, and the United Kingdom, with the top three institutions in policy publication volume remaining the same as in the first generation. Policy categories such as Medical and Health Sciences, Biological Sciences, and Law and Legal Studies exhibit the highest citation frequencies for NPs, first-generation citations, and second-generation citations in gene engineering.

In astrophysics, policy documents within the field of Information and Computing Sciences published by the U.S. National Academies Press cite NPs. For first-generation citations, documents from the United States, Switzerland, and Italy collectively account for 96%, with the World Meteorological Organization, Center for Strategic and International Studies, and Food and Agriculture Organization of the United Nations being the top three institutions in terms of policy publication volume. For second-generation citations, the United States, Switzerland, and Luxembourg publish the highest number of citing policy documents. At the institutional level, the Publications Office of the European Union, World Meteorological Organization, and Analysis & Policy Observatory are the top three institutions in terms of policy publication volume. In terms of policy categories, those citing first-generation citations mostly fall under Physical Sciences, Information and Computing Sciences, and Earth Sciences, while those citing second-generation citations mainly belong to the categories of Physical Sciences, Studies in Human Society, and Earth Sciences.

All citing policy-publishing institutions can be categorized as international, national, or regional, with one exception: only national institutions cite NPs in astrophysics. We also calculated the Gini index for the distribution of citing policies across countries, organizations, categories, and so on.

We explored the policy impact of Nobel Prize-winning papers (NPs) and their citation generations. Figure 5 displays the distribution of policy citations for NPs and their generations, while Figure 6 illustrates the differences. On average, gene engineering NPs have 0.3378 policy citations, with 1st and 2nd generations having 0.0234 and 0.0141, respectively. In astrophysics, these values are 0.0203, 0.0013, and 0.0026.

Figure 5.

Figure 6.

Negative binomial regression was applied to control for potential confounders that effect policy citations. To avoid correlation among variables, we also performed multicollinearity tests by VIFs. The results in Table 5 show that there were no serious problems of multicollinearity.

The regression results, as shown in Table 6, indicate that in gene engineering, NPs have a significantly higher policy impact than the first generation of citations, which, in turn, has a higher policy impact than the second generation of citations (Table 6, column 1). We further ensured the robustness of these results by controlling for confounding variables: Publication type, N-research countries, N-publication citation, N-references, and Publication age (Table 6, columns 2-7). In all cases, the intergroup difference between NPs in gene engineering and their citation generations in policy impact remained consistent, demonstrating the robustness of our findings.

However, for NPs in astrophysics, the results in Figure 6b and Table 6, columns 8-14, indicate no significant difference in policy impact between NPs and their citation generations.

Furthermore, Table 6 highlights that in gene engineering, policy impact is positively correlated with N-research countries, N-publication citations, and N-references. In contrast, in astrophysics, policy impact is primarily associated with N-publication citations.