Question: How do I test the difference between two diff-in-diff coefficients?

[David Zhong]

Hi Statalist,

I want to test whether a normal MA event will have different effect on the outcome from a PEVC event. The regression is
outcome = Post*PEVC + Post*MA + controls+ FEs+ e

reghdfe outcome c.post#c.PEVC c.post#c.MA $controls, absorb(cohort#firm cohort#ind#year) cluster(cohort#firm)

then I use

test _b[c.post#c.PEVC] = _b[c.post#c.MA]

Is this a correct way to test?

Best,
DZ

[Clyde Schechter]

What you have shown is a correct way to test equality of the coefficients. But from that you cannot infer whether the two interventions have different effects, or which is greater. In fact, unless MA and PEVC are measured in the same units, on the same scale, and have the same overall distributions, there is no way to compare the effect of one to the other. More or less the only circumstance in which the conditions for making such a comparison are met in the real world is when each of the interventions is a dichotomous variable, and both events occur with the same frequency.

[David Zhong]

What you have shown is a correct way to test equality of the coefficients. But from that you cannot infer whether the two interventions have different effects, or which is greater. In fact, unless MA and PEVC are measured in the same units, on the same scale, and have the same overall distributions, there is no way to compare the effect of one to the other. More or less the only circumstance in which the conditions for making such a comparison are met in the real world is when each of the interventions is a dichotomous variable, and both events occur with the same frequency.

Thanks for your explanation! And yes, both variables are dummy variable which equals 1 if it is in the treat group (PEVC or MA), and the control group consists of all firms and their detailed units that have never been treated.
At the detailed level, I do find PEVC have significantly different coefficient from MA, like this:


| Robust
log_cal | Coefficient std. err. t P>|t| [95% conf. interval]
---------------------+----------------------------------------------------------------
1.intpevc | -.0448502 .0228433 -1.96 0.050 -.0896239 -.0000765
1.intma | .1162355 .0297535 3.91 0.000 .0579175 .1745536

And the Fstats is 18.04

However, at the firm level, most cases are like:

| Robust
log_cal| Coefficient std. err. t P>|t| [95% conf. interval]
---------------------+----------------------------------------------------------------
1.intpevc | -.0910865 .053559 -1.70 0.089 -.1960653 .0138924
1.intma | .0401645 .1387508 0.29 0.772 -.2317952 .3121243
and the Fstats is like 0.79.

Can I still say PEVC has more significant effect on the outcome variable than MA. This feels wrong since the Fstats is too small.

Best,
DZ

[Clyde Schechter]

I'm sorry, but I do not understand what you mean when you refer to the "detailed level" and the "firm level." It appears you have done two different analyses, but you haven't provided enough information about what those analyses were. Also your reference to F statistics is unclear: are these the overall F-statistics of the regressions or are they the F-statistics of the -test- commands that you ran post-regression?

Let me also reiterate that in order for the difference in coefficients to be interpretable as a difference in effect, the frequencies of the two occurrences must be (approximately, within reason) equal. You have not shown your results with respect to that criterion, so I remain agnostic (skeptical) of the usefulness of testing the equality of the coefficients.