Diff-in-diff with count dependent variable: OLS vs Poisson vs Negative Binomial

[Paula de Souza Leao Spinola]

I am estimating a diff-in-diff simple model with an interaction term consisting of the product between a dummy of treatment status and a dummy of post-treatment time period. The questions I have concern the fact that my dependent variable is a count variable.

I am not sure whether I should use Poisson or Negative Binomial regressions instead of OLS. I tried estimating the three models. Then, using the margins command, I computed the mean outcome difference between treated and controls units for both the baseline and the post-treatment periods.

Strangely, the marginal effect estimates for the Poisson specification are identical to those obtained by the OLS regression. Is this reasonable? The Negative Binomial marginal estimates are also very similar to the other two models.

The mean predicted outcome values from the three models are very similar to that of the observed data and are also never negative (as in the original data). However, they are also never equal to zero and have a much narrower range. The three models predict outcomes ranging from 7 to 28 while the observed values range from 0 to 504.

I am not sure how to proceed.

* ~~~ OLS
reg outcome treat##pos, cluster(health_cent_id) 
predict yhat_ols
margins pos, dydx(treat) nopvalues
/*
--------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
0.treat      |  (base outcome)
-------------+------------------------------------------------
1.treat      |
         pos |
          0  |   .6509491   2.337706     -3.971704    5.273603
          1  |   11.18207   3.615578      4.032511    18.33162
--------------------------------------------------------------
*/

* ~~~ Poisson
poisson outcome treat##pos, cluster(health_cent_id) 
predict yhat_poisson
margins pos, dydx(treat) nopvalues
/*
--------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
0.treat      |  (base outcome)
-------------+------------------------------------------------
1.treat      |
         pos |
          0  |   .6509491   2.337107     -3.929696    5.231594
          1  |   11.18207   3.614651      4.097482    18.26665
--------------------------------------------------------------
*/

* ~~~ Negative Binomial Model
nbreg outcome treat##pos, cluster(health_cent_id) 
predict yhat_nb
margins pos, dydx(treat) nopvalues
/*
--------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
0.treat      |  (base outcome)
-------------+------------------------------------------------
1.treat      |
         pos |
          0  |   .6509489   2.337107     -3.929696    5.231594
          1  |   11.18207    3.61465      4.097481    18.26665
--------------------------------------------------------------
*/
su outcome yhat_ols yhat_poisson yhat_nb if e(sample)==1
/*
    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
     outcome |      5,852     13.5716    27.74452          0        504
    yhat_ols |      5,852     13.5716    7.877809   7.769357   28.34767
yhat_poisson |      5,852     13.5716    7.877809   7.769357   28.34767
     yhat_nb |      5,852     13.5716    7.877807   7.769355   28.34766
*/

[Jeff Wooldridge]

You're not computing the correct marginal effect. When you have a saturated model -- as is always the case in the 2 x 2 DiD -- the fitted values will be the same no matter what. What using an exponential model does is change the underlying parallel trends assumption, and so the average treatment effect on the treated is not obtained by applying the margins command to treat. The actual treatment variable is treat*post.

To get the average treatment effect on the treated for Poisson -- preferred over NegBin for robustness -- do the following:

gen w = treat*post
poisson outcome treat post i.w, vce(cluster health_cent_id)
margins, dydx(w) at(treat = 1 post = 1)

This is comparable to the coefficient in the linear regression. I discuss these issues in my 2023 Econometrics Journal paper on nonlinear DiD. Also, I have a shared Dropbox with more examples:

Nonlinear Did

[Paula de Souza Leao Spinola]

Thanks a lot Prof. Wooldridge - what a privilege to receive a response from you!

I understand that the interaction term is the main coefficient of interest in a diff-in-diff model.

In reality, the intervention I evaluate is an RCT. For this reason, I could have regressed the outcome variable on the treatment dummy using post-intervention data. However, given that I have pre-intervention data, I included them in the model and, therefore, specified a diff-in-diff regression model. I would like to show that while there are differences in the outcome during the post-policy period, there was no difference before the start of the intervention. For this reason, I am interested in estimating the margins command below. This informs the difference between the treatment and control groups before and after the intervention.

margins post, dydx(treat)

Does this make sense? Or would you suggest I ignore the baseline data and, instead, run a simple regression of the outcome on the treatment assignment indicator?

Thank you very much.
Paula

[Kehinde Atoloye]

You're not computing the correct marginal effect. When you have a saturated model -- as is always the case in the 2 x 2 DiD -- the fitted values will be the same no matter what. What using an exponential model does is change the underlying parallel trends assumption, and so the average treatment effect on the treated is not obtained by applying the margins command to treat. The actual treatment variable is treat*post.

To get the average treatment effect on the treated for Poisson -- preferred over NegBin for robustness -- do the following:

gen w = treat*post
poisson outcome treat post i.w, vce(cluster health_cent_id)
margins, dydx(w) at(treat = 1 post = 1)

This is comparable to the coefficient in the linear regression. I discuss these issues in my 2023 Econometrics Journal paper on nonlinear DiD. Also, I have a shared Dropbox with more examples:

Nonlinear Did

Please, is this applicable to probit model?

[Jeff Wooldridge]

Yes, but for reasons I describe in my 2023 Econometric Journal paper, logit has a slight advantage. Probably not enough for it to make much difference, though.

logit outcome i.w treat post, vce(cluster id)
margins, dydx(w) at(treat = 1 post = 1)
margins, dydx(w) at(w = 1)

The two margins commands are the same. If you have more time periods and/or a staggered intervention, it's a bit more complicated. See here:

Shared Dropbox Folder

[Kehinde Atoloye]

Yes, but for reasons I describe in my 2023 Econometric Journal paper, logit has a slight advantage. Probably not enough for it to make much difference, though.

logit outcome i.w treat post, vce(cluster id)
margins, dydx(w) at(treat = 1 post = 1)
margins, dydx(w) at(w = 1)

The two margins commands are the same. If you have more time periods and/or a staggered intervention, it's a bit more complicated. See here:

Shared Dropbox Folder

Thanks a lot Prof. Wooldridge

[Kehinde Atoloye]

Yes, but for reasons I describe in my 2023 Econometric Journal paper, logit has a slight advantage. Probably not enough for it to make much difference, though.

logit outcome i.w treat post, vce(cluster id)
margins, dydx(w) at(treat = 1 post = 1)
margins, dydx(w) at(w = 1)

The two margins commands are the same. If you have more time periods and/or a staggered intervention, it's a bit more complicated. See here:

Shared Dropbox Folder

Dear Prof,

Please, for this model, if one wants to plot this type of graph, what particular information from the model would one have to use. This graph was plotted from a linear regression DiD but since this model here is a logit model, I am thinking there might be a need for some adjustments since "the parallel trend assumption may not be met". Will the presence of covariates also play a role?