Guidance Regarding Interpretation of Difference-in-Difference (DID) Estimations

[Ammar Hussain]

Hi, Respected Researchers.

For Example my topic is to checks the impact of high leverage on earnings management (EM). EM is dependent variable and high leverage is our treatment variable equals 1 if leverage is greater than its 50th percentile value and 0 otherwise. Further, I also test whether this relationship is changed after the change is debt policy 2015. So, I generate Post variable equals 1 if years after 2015 and 0 for years before 2015.
Note: I have unbalanced Panel data from 2002 to 2022 of more than 2000 firms.

For DID Estimations, if I regress the command in the following manner:

regress EM Treatment c.Treatment#c.Post CONTROLS, vce(cluster id)

then get the following results:
DV= EM

Variable	Coefficient	P-Value
Treatment	0.006477	0.001
TreatmentPost	0.003241	0.044
Controls...	...........	..........

Please guide 1) how I interpret these results? what is the impact of policy time intervention on EM when use interaction of Treatment and Post variable, because both coefficients are positive and significant but in form of interaction 2nd coefficient is almost half than Treatment coefficient. Please explain how interpret this. 2) is this method is correct for DID estimations of panel data? because i dont get good results and get some error when use xtdidregess function.

Further, for the conciseness, not report the results of all 8 control variables.

@Clyde Schechter @Roman Mostazir @Rivanka Diaryzki @Sebastian Schirner @George Ford @all_other_experts

Thanks

[Ammar Hussain]

Same in Case, if I use reghdfe command for above equation, like:

reghdfe EM Treatment c.Treatment#c.Post CONTROLS, absorb(year) vce(cluster id)

Then find the following Results:

hen get the following results:
DV= EM (1st Proxy)

Variable	Coefficient	P-Value
Treatment	0.0477	0.001
TreatmentPost	- 0.03241	0.178
Controls...	...........	..........

With 2nd EM Proxy

Variable	Coefficient	P-Value
Treatment	0.0388	0.001
TreatmentPost	0.1324	0.222
Controls...	...........	..........

In short, in both proxies the Treatment effect is positive and significant but main DID estimator is insignificant in both proxies but in 1st case coefficient is negative and in 2nd case coefficient is positive. If I use one of from these then how I interpret the results.

[Clyde Schechter]

Your first model using -regress- is incorrect because it does not include the post variable as a "main" effect. It has only the treatment group variable and its interaction with post. Your -reghdfe- model overcomes that limitation by absorbing the year variable.

Here's a toy example of different models:

webuse grunfeld, clear

summ year, meanonly
gen byte prepost = year <= r(mean)
summ company, meanonly
gen byte treatment = company <= r(mean)

regress mvalue i.treatment##i.prepost

reghdfe mvalue i.treatment#i.prepost, absorb(company year)

reghdfe mvalue i.treatment i.treatment#i.prepost, absorb(year)

regress mvalue i.treatment i.treatment#i.prepost

Note that the first three models, which incorporate a treatment or company effect, and a prepost or year effect separate from the interaction term, either explicitly or through absorption, all give the same estimate for the treatment effect (interaction term coefficient). (They differ in standard errors, but the estimates are the same.) But the fourth one, which is analogous to your -regress- command, gives a different value. The first three are all correct, and the last is not.

[Ammar Hussain]

Respected Clyde Schechter Sir, Thanks for highlighting my mistake about regress command. Further, if my -reghdfe- command is correct then how I interpret the "Treatment" and "TreatmentPost" coefficients under both EM proxies. What does "TreatmentPost" insignificant coefficients shows (in one case negative and in one case postive)? while treatment coefficient is positive and significant in both cases. https://www.statalist.org/forums/for...39#post1742839
Please kind the interpretations in this regards, whether the policy intervention weakend this relation or not effects the relation?


Thanks again.

[Clyde Schechter]

The coefficient of the TreatmentPost interaction term is the DID estimator of the treatment effect. The coefficient of Treatment by itself is just the expected difference in value of EM between the treatment and control groups in the pre-treatment era, and may or may not be of any interest. In any case, the coefficient of Treatment, notwithstanding its name, tells you nothing about the treatment effect. If it has any role in interpreting the result, it is for secondary or contextual purposes only.

As for the analyses using proxy outcomes, there is nothing particularly surprising about these results. You don't show the confidence intervals for these, so I can't be more specific, but since neither of those results is even close to statistically significant, we can conclude that the data are consistent with either positive or negative treatment effects on both proxy outcomes. A more succinct way to say that is that the analyses using proxy outcome variables are simply inconclusive. They do not tell you much about the treatment effect at all, not even its direction. If these proxies have previously been widely used and are considered to be good, strong proxies for EM itself, then it raises questions about the design of your study or whether your data contain correct values for them. If these are proxies which are not widely used, or are considered questionable, then there is nothing unusual about getting inconclusive results with them.

[Ammar Hussain]

Thanks for guidance, above results are just hypothetical not actual, just shows the actual nature positive or negative / sig or nonsig, otherwise results are hypothetical. Thanks for guidance .

[George Ford]

While you can get a coefficient, this case is bit strange.

If you think of your treatment as US policy, then it applies to everyone. There is no DID available. If you think of your treatment as high leverage, then you don't necessarily have a change in 2015, and the treatment is not exogenous but the choice of firms.

There is a DID between high-low leverage in two policy states (which apply to all firms), but I'm not sure whether this is a DID in the normal sense. It's certainly not a clean 2x2 DID. Maybe someone smarter than me will jump in and correct me.

Also, you've arbitrarily cut a continuous variable into a dichotomous one, which isn't ideal. Typically, a debt/asset ratio > 1 is considered high (which you could say is an exogenous cut; I'd definitely find something from standard practice regarding high leverage), but I'm not sure if your definition allows you to use that cutpoint. I'd definitely play around with that (maybe divide leverage into a few groups), if you use it at all.

[Ammar Hussain]

Thanks for Explanation, above I just show hypothetical scenario for understandings, In actual our Treatment variable is Climate Change Exposure if its value is greater than 60th percentile then equals 1, otherwise 0.

Now, just guide me in one matter, Now I use the following command for DID estimates and got the results as per our expectations:

reghdfe WC_ACC_dec Treatment TreatmentPost Control1 Control1 Control2 Control3 Control4, abs(year) vce(cluster id)

Please tells me, whether I write the above equation in right format for DID estimate or not?

[Clyde Schechter]

Yes, it is suitable. If I were doing this myself, I would also absorb the id variable. That will change the standard errors, but not the effect estimate. It will also cause the Treatment variable to be omitted due to colinearity with the absorbed id variable, but that is not a problem because the Treatment coefficient is of no importance anyway and is just a meaningless artifact.

[Maxence Morlet]

Dear Ammar, it seems like your treatment is in fact a continuous variable. You may thus want to read the following papers: “Event Studies with a Continuous Treatment” and “Difference in Difference with Continuous Treatment”. Both are NBER papers

[Ammar Hussain]

Thanks alot Maxence Morlet and Clyde Schechter

[Ammar Hussain]

Hello,
Below are the two examples of rddensity plotting. As per my understandings, this test is used to checks the manipulation of running variable around the cutoff value in regression discontinuity design (RDD) analysis. If the p-value > 0.05, its means there is no systematic manipulation around cutoff (same like both below figures p-value fairly >0.05).

My question is that in 1st figure right side line is below than left, while in 2nd image right side line is above than left side. To enhance my understandings, I want to knows that is these lines depicts something in rddensity plotting? and this left and right lines mismatch in both figures reflects something different interpretation in both figures(or just to show discontinuity) while p-value is insignificant in both cases., or P-value is enough to interpret rddencity plotting?


Thanks