Difference in Differences Result

[Jesita Wida]

Hey everyone! My name is Jesita. I am currently writing a paper regarding the impact of education decentralization on students' standardized test scores using IFLS panel data from 1993 to 2017 with 9,141 observations.
This is my result:

xtreg std_score1 i.MoEC i.Post i.MoEC##i.Post, robust cluster (commid) fe i(fcode)

Fixed-effects (within) regression Number of obs = 9336
Group variable: fcode Number of groups = 4800

R-sq: within = 0.0062 Obs per group: min = 1
between = 0.0114 avg = 1.9
overall = 0.0031 max = 5

F(3,313) = 6.55
corr(u_i, Xb) = -0.1643 Prob > F = 0.0003

(Std. Err. adjusted for 314 clusters in commid)
------------------------------------------------------------------------------
| Robust
std_score1 | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.MoEC | .0119933 .0575029 0.21 0.835 -.1011477 .1251343
1.Post | .2059144 .0610805 3.37 0.001 .0857341 .3260946
|
MoEC#Post |
1 1 | -.2950334 .0673567 -4.38 0.000 -.4275625 -.1625043
|
_cons | -.0060922 .0395492 -0.15 0.878 -.083908 .0717237
-------------+----------------------------------------------------------------
sigma_u | .94483298
sigma_e | .82235053
rho | .56897805 (fraction of variance due to u_i)
------------------------------------------------------------------------------

The conclusion is that: "decentralization negatively affect students' test scores"
I ran the balanced panel and full sample analysis, with or without controls, and the results are consistent.
My biggest question: How do I know that the result is robust and correct? I know I cannot use "diff" command because my data is panel.
Please help me. Thank you so much.

[Clyde Schechter]

How do I know that the result is robust and correct?

You don't. All you can ever say from this kind of analysis is that you have correctly implemented a difference in differences analysis and this is what it found. There remain the issues of whether the underlying assumptions of the DID strategy for identifying causal effects are plausible.

One of those you can test: the parallel trends assumption. You can plot the data year by year during the pre-decentralization era in both groups and observe whether they trend in parallel.

Another key assumption, which you cannot test, is that the decentralization is the only thing that happened during that implementation year that affected the test scores. The analysis demonstrates that in the year that decentralization began, test scores increased by approximately 0.21 in the group that was not subjected to decentralization, whereas in the group that did undergo decentralization, scores fell by about 0.09. But if there were other things happening that year which might also have affected the scores and affected the decentralized and non-decentralized entities differently,, the analysis cannot distinguish the effects of those other things from the effects of decentralization. Your data will not have any information about that. The best you can do is try to learn what other things were happening then that might have made a difference.

You can also check the robustness of the analysis by doing some sensitivity analyses. Change the definition of the Post variable to use some different cutoff years. If it's really the decentralization that is at work, then you should get negative results for other years. (But, be cautioned: there can sometimes be anticipatory effects, or delayed effects, of the intervention itself that show up when you look a short amount of time before or after the actual implementation date.) You can also do a randomization test: replace the treatment group variable by a random assignment: again, you should get negative results.

But no matter how many tests and checks you do, you will never achieve certainty. Nobody can.

[Jesita Wida]

Thanks Clyde!