Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

Assume individuals in a set of observational data as I. i∈I is one of the individuals. Y is the outcome variable in direct relationship to the intervention event. Denote t is the timestamp of the observed period in any kind of unit of time. T is the time dummy variable that equals 0 when t≤0 and 1 when t>0. D is a dummy variable indicating whether individual i is treated, D=1 for individuals of the treatment group, and D=0 for those of the control group. The cross term T_t×D_i equals 1 only for the posterior observed outcomes of treated individuals. Covariates X_k (k=1, …, K) are confounders that might influence the dependent variables. They are not asked to be time-varying, for example, in the estimation of earthquake's causal effect on the GDP of cities, covariates could be the locations of cities. Specifically, the potential outcome of an individual i exposed or not to a treatment is given by Y_iT=D_iY_iT(1)+(1–D_i)Y_iT(0), where Y_iT(1) denotes individual i's potential outcome in period T if i is untreated in the first period but exposed to treatment by the second period and Y_iT (0) denotes individual i's potential outcome in period T if i is untreated in both T=0 and T=1 period.

Before using the DID regression design, three fundamental premises must be met. The first is the Stable Unit Treatment Value assumption (SUTVA) (Rubin, 1977). SUTVA implies that the outcomes for everyone at any time from observational data are unrelated to any other individuals’ treated status. The second is to assume that covariates are not affected by the treatment, i.e., the exogeneity is satisfied. Especially in DID, there is one additional assumption that treatment is only active after the treatment time and has no effect on the pre-treatment population (NEPT).

The most important assumption is the common trend assumption (CTA, also known as the parallel trend assumption).

Common trend assumption (CTA): E(Yi1(0)−Yi0(0)|Di=1)=E(Yi1(0)−Yi0(0)|Di=0) {\rm{E}}\left( {{Y_{i1}}\left( 0 \right) - {Y_{i0}}\left( 0 \right)|{D_i} = 1} \right) = {\rm{E}}\left( {{Y_{i1}}\left( 0 \right) - {Y_{i0}}\left( 0 \right)|{D_i} = 0} \right)

CTA is a core assumption in DID regression design to impute the mean untreated outcomes for the treated group by using the change in outcomes for the untreated group and the baseline outcomes for the treated group. Based on the CTA, it could be stated that the average difference in outcome between the treated and control group remain relatively steady before treatment occurred (Roth et al., 2022). It is so-called parallel rationally therefore that CTA asks for no difference before and after the treated time of untreated units and treated units if they are untreated (See Figure.1). Because the potential outcome of treated individual i when i is untreated is unobservable in practice, the left-hand side of this assumption yields the counterfactual of the trend which does not exist.

Figure 1

Another important and often hidden assumption required for the identification of ATT is the no-anticipation assumption, which states that the treatment has no causal effect before its implementation.

No anticipatory effects assumption (NAE): Yi1(0)=Yi1(1)∀ i s.t. Di=1. {Y_{i1}}\left( 0 \right) = {Y_{i1}}\left( 1 \right)\forall \,i\,s.t.\,{D_i} = 1.

The strong CTA is a well-known limitation of the Difference-in-Differences Method. Figure 1 simply tells the shape of trends when the CTA is satisfied in an observational study. Finding a control group whose trend of outcome is nearly parallel to that of the treatment group before the therapy might be challenging. The optimal scenario is one in which the dependent variable has been well covered by the manually created model and the intervention is the only factor responsible for the difference between the two groups. In practice, the CTA is tested empirically in the procedure of designing a regression model. We can add time and treatment interaction dummy variables before the intervention and evaluate their coefficients. Given that they are not significant and the estimated 95 percent confidence intervals of them cover 0, we could conclude that the difference between comparative groups is insignificant, i.e., trends in the outcome of treatment and control groups are parallel. The CTA, otherwise, is violated if the coefficients of pre-treatment interaction terms are significant, which will result in a biased and invalid estimation.

The idea of CEM is first proposed by Iacus et al (2012), which is one of the simplest methods with monotonic imbalance bounding (MIB) properties. The underlying principle of CEM is to coarsen each variable through recoding so that practically indistinguishable values are grouped and given the same numerical value (for example, a continuous variable with numerous distinct values is coarsened into a prespecified number of categories, such as quintiles). Then, according to the coarsened data, the “exact matching” algorithm is used to find matches and prune unmatched groups. The coarsened data are removed, and the original information of the matched data is finally retained.

The CEM algorithm generates a collection of strata, and units in strata that contain at least one treatment, and one control unit are kept in this sample, and units in the remaining strata are removed from this sample. Several criteria and algorithms may be used to create and choose strata. The quantile approach and manual division based on prior experiences are the two most used coarsening techniques. For instance, there are several covariates to consider if we wish to compare a Nobel laureate's scientific innovation to that of a non-prizewinning nominee. These covariates include the research field, career age, publications, and citations, among others. The research field is used as the coarsened variable in this instance; each field group has at least one laureate and several non-prizewinning nominees. The exact matching approach was then used to match the non-prizewinning nominees for laureates. Therefore, even if it is not a perfect match, the treatment field has a match.

Eliminating all imbalances is the task that CEM puts to matching (i.e., differences between the treated and control groups). All multivariate nonlinearities, interactions, moments, quantiles, and other distributional differences above the selected threshold of coarsening are among the imbalances that CEM eliminates. Since the remaining differences all contained small, coarsened strata, a statistical model might be used to bridge them with little risk of model dependence.

The other famous matching algorithm is PSM, first introduced by Rosenbaum and Rubin (1983). Use the notation yielded in the DID regression model. The performance of PSM has been extensively examined via simulations and theoretical demonstration (Abadie & Imbens, 2006; Austin, 2009). The potential outcome is a variable of interest with covariates X irrespective of the treatment. The conditional independence assumption is to be ensured to eliminate the selection bias. It describes that potential outcomes are independent of treatment assignment when conditioned on the covariates. Another vital assumption is the common support. It rules out the phenomenon of perfect predictability of D given X: 0 < P(D = 1|X) < 1. It ensures that persons with the same X values have a positive probability of being both treated and control individuals (Heckman et al., 1999). Given these two assumptions satisfied, the PSM could be used to estimate the ATT over the common support of treated and control groups (Becker & Ichino, 2002).

Look into the details of PSM, and the canonical steps including choosing an appropriate subset of covariates X. Then the propensity score of everyone is calculated by estimation of the probit or logit regression model. After this step, each individual is assigned a value ready for matching (Coca-Perraillon, 2007). There are several matching algorithms that act as classification estimators including nearest neighbor (NN) (Cover & Hart, 1967), radius matching (Huber et al., 2015), interval matching (Hill & Reiter, 2006), kernel matching and other matching techniques (Garrido et al., 2014). The parameter that received the most attention in evaluation literature is also the average treatment effect on the treated group (ATT) as defined in the former section. In dealing with observational data problems on outcomes independent of time, PSM can estimate the ATT of outcomes and reach conclusions.

For time-dependent outcomes in DID regression models, PSM could only act as a pre-match. In recent years, there has been a trend of natural experiments in the study of economics, management, and medicine. Relevant statistical models have been developed rapidly, among which, PSM-DID is a powerful policy analysis tool. The PSM-DID model is a combination of the propensity score matching model (PSM) and the DID model. Here PSM contributes to the identification of control group individuals as similar as possible to the treatment ones and the DID regression is responsible for the evaluation of the effect caused by treatment. PSM-DID can control individual differences not only generated by observable covariates but also by unobservable covariates when compared with PSM. More than DID, the PSM-DID model takes individual differences into account between the treated group and the control group.

Both CEM and PSM can be taken in priority for more precise and valid estimation results from the DID regression. When the number of observations is too large for PSM to calculate the difference in propensity scores between the treated unit and other control units. In this sense, CEM, as a non-parametric method, performs better than PSM especially when the matching purpose is to find N most similar control units for N treated units, i.e. an N:N match (Wang, 2021). Simple PSM should be avoided unless necessary, such as greedy 1:1 matching without replacement, which matches treated subjects in any order and determines for each one the best match among the unmatched controls. It is advised that researchers carefully plan and design PSM, assess its impact with the knowledge at hand during modeling preparation, and set caliper width considering matching and population biases beforehand (Stuart, 2010).

In this practice, we are trying to evaluate the status spillovers of prizewinning—how winning a scientific prize affects the winner's future scientific performance compared to her/his collaborators who coauthored the prizewinning paper. For a long time, scientific prizewinning is regarded as a halo for winners, so the winners will benefit a lot in terms of reputation, funding, impact, etc. in the future (Borjas & Doran, 2015; Chan et al., 2014; Merton, 1968). The key to quantitatively evaluating the “halo” effects in future citation impact is to have the proper comparison group. Winners are a special group that has great contributions to science, here we select the winner's collaborators who coauthored the prizewinning works with the winner as the comparison group, so that both the winners and the coauthors have the prizewinning contributions to science, the difference is that one group won the prize and the other group did not. We first used the DID method to estimate the treatment effects, then to further eliminate the other confounding factors like productivity, we applied the CEM and PSM matching before the DID to improve the estimations.

We used the Nobel prize dataset collected by Li et al. (2019) which provides details on the 545 Nobel laureates in the disciplines of physics, chemistry, and physiology/medicine, including their full list of publications and their representative prizewinning works we then extended the dataset to get all the laureates’ coauthors who collaborated the prizewinning works and all scientists’ yearly based citations, by using the OpenAlex (https://openalex.org/) corpus.

Our comparison is based on two groups of scientists. According to Nobel prizewinning papers published ranging from 1887 to 2010, the treatment group is made up of Nobel laureates in the field of physics, chemistry, and physiology or medicine with winning years ranging from 1902 to 2016, while their coauthors working in the prizewinning papers are assigned to the control group. We assume that teammates of a prize-winning work share similar capacities and be possessed the same potential to make a breakthrough in their research areas. There are a total of 2,010 scientists in the control group and 635 in the treatment group. To verify our findings, winners who persistently publish papers ten years before and after the year when their studies won the Nobel prize were chosen. Prizewinning works written by a single author are ignored. Eventually, there are 335 laureates and coauthors in total. Table 2 shows the descriptive statistics for our data.

Denote citation impact of a scientist i at a certain year t is C_it. As citations are usually left-skewed distribution (Radicchi et al., 2008), we adopt the rescaled citation impact c_it=log(C_it+1). In the first part of modeling, we use the DID regression to calculate the average treatment effect of rescaled citation impact c_i for the treated group (ATT), E(c_i|D=1, T=1)–E(c_i|D=1, T=0), which is estimated through the regression model: (4) cit=β0+β1*Treati*Postt+μi+τt+εit, {c_{it}} = {\beta _0} + {\beta _1}*{Treat_i}*{Post_t} + {\mu _i} + {\tau _t} + {\varepsilon _{it}},

Treat_i is a dummy variable indicating if scientist i is the Nobel laureate or not, it equals 1 when i is the winner and 0 when i is the winner's coauthor. Post_t equals 1 when t>0 and 0 otherwise. There are individual fixed effect variable μ_i and time fixed effect variable τ_t, so the sole dependent variables Treat_i and Post_t are omitted because they are colinear with fixed effect variables and the coefficients β₁ of their cross term Treat_i*Post_t are the ATT we want to estimate. In this problem, we then use the DID regression module in Stata (Donald & Lang, 2007; McCaffrey & Bell, 2003).

According to our selection of the control group, the winner's prizewinning collaborators are supposed to be possessed similar scientific potentials and capacities, then are comparable to the winners before the prizewinning year. So, at first, we could straightforwardly perform a DID regression model on the outcome Y=c_i. The result is in table. We can conclude that the difference between the treated group and the control group after the prizewinning years is insignificant, which means that the Nobel prizewinning may not improve the winner's future citations.

However, we found that there is an issue that the common trend assumption of DID is not satisfied. This implies that simply selecting the coauthors of winners is unable to eliminate the effect of the potential endogeneity. One important factor is the different productivities between winners and the coauthor counterparts. In this regard, we used the CEM and PSM to identify coauthors who had been working on their research for a comparable amount of time which is like the winners, i.e., we need to consider their production rate in careers. In detail, t₀=0 is the Nobel prizewinning year. Select t₁= −10, t₂= −5, the matching covariates (number of papers at year t) used in PSM and CEM methods are p_it1, p_it2, p_it0 To maintain the common trend assumption in DID, the average publication in ten years before winning is used, which is numerically defined as p¯i=111∑t=−100pit {\bar p_i} = {1 \over {11}}\sum\nolimits_{t = - 10}^0 {{p_{it}}} . The productive capacity of scientists will affect their citation impact to a great extent, so these four covariates are selected for representing the productivity of scientists before the winning year in this experiment. The matching techniques then are applied to find coauthors with similar productivities in the 10-pre-prizewinning years to laureates.

We used Stata to conduct PSM (Leuven & Sianesi, 2003) and CEM (Blackwell et al., 2009) respectively. A k-nearest neighbor algorithm is used in the PSM matching process to offer a matching individual for each treated individual. The coauthors from the PSM-matched control group have the similar productive capability to winners in the ten years prior to prizewinning. Further, we utilize a CEM method on the same set of covariates. Table 3 shows that the pre-treatment citations are becoming closer after we applied the CEM and PSM methods.

As shown in Table 3, using the CEM and PSM substantially eliminated the pre-gaps between treatment groups and control groups. Only 80 pairs of treated and control units with comparable pre-academic productivities remain after CEM, which abandons treated units when matching control units cannot be discovered. For each treatment unit, PSM matches the most comparable control units. The disparities in productivity between the two groups are reduced by these two matching procedures, which also minimizes the promiscuous effects of productivity on citation impacts. Finally, we summarized the results of three DID regression models in Table 4.

Given that the CTA is met in Models II and III after combining CEM and PSM with DID, respectively, Table 4 reasonably infers that scientists who have won the Nobel Prize do not witness a noticeable increase in their citation impact as compared to their prizewinning coauthors. When laureates receive Nobel awards, their reputations unquestionably improve in a variety of ways, one of which is an increase in citation impact. However, the prizewinning papers were often works of laureates a long time ago, these milestones’ huge potentials had already been recognized and their era significance in pushing the scientific development process had been widely acknowledged. From this perspective, it is not surprising that their citation impacts did not grow after their victory.

In this work, we demonstrated how to use the DID method in science of science studies and how to use the common statistical matching approaches such as PSM and CEM to improve the estimation. We gave detailed definitions of the DID method, and the mathematics behind it we emphasized the conditions before applying the DID method and introduced several matching methods to improve the estimation. As a case study, we studied how winning a scientific prize affects the winner's future scientific performance compared to her/his collaborators who coauthored the prizewinning paper.

The underlying premise of DID is that, in the absence of therapy, the average outcomes for the treated and control groups would have progressed similarly over time. The parallel trends assumption, however, is unrealistic in many health policies, economic, or science of science settings because of potential confounders.

Consider scenarios where the common trend assumption holds, the estimation is unbiased, and the DID estimator provides a clear estimate of the event's expected causal effect. On the other hand, the predicted values should be false positives anytime the CTA is broken because confounding bias exists. In this regard, it is necessary to modify the settings, uncover additional potential factors, and ensure that the CTA is valid before performing the DID regression.

To reduce any potential imbalance between the treatment and control groups, we present two matching strategies in this paper: CEM and PSM. Both alternatives can be used to maintain the CTA. CEM is a nonparametric matching technique that can be used in a quasi-experimental design to compare an outcome over time between two groups. When compared to PSM and used in the context of a comparison group framework, CEM has been demonstrated to produce estimates of the causal effect with lower variance and bias for any sample size (King et al., 2011).

We recommend researchers test both CEM and PSM simultaneously and assess how well they choose the proper control group units for the treatment ones in the CTA test of DID regression. However, PSM is not advised to be used when the size of the control group is significantly bigger than that of the treated group because this will prevent the propensity scores from being calculated due to inadequate processing capability (King & Nielsen, 2019). Especially for 1:1 matches, the CEM performs better than PSM (King et al., 2011). The PSM is advised if we wish to locate control units for each treatment unit without missing any of them because of matching failures. While in CEM, treated units may be abandoned if they were not in some stratum with control units, it would match one or more control units as similarly as feasible to a treated unit.

In this section, we summarize the most crucial aspects of using matching techniques in studies that want to apply DID in science of science studies.