DiD coefficient is insignificant for country FE but adding country and time FE changes coefficient sign and becomes significant?

[Mathew Chandy]

I'm looking at whether an environmental policy in 2011 had an affect on exports for the UK using difference-in-differences (Australia is my control). When I added country fixed effects to the equation, I got a positive coefficient for my treatment variable but it was insignificant. When I added time fixed effects as well, it gave me a negative coefficient that was significant at 1%. I am struggling to come up with a correct and detailed interpretation of these initial results - any help would be appreciated!

Exports = β₀ + β₁y2011 + β₂country + β₃y2011*Country + ε
(Basic diff-in-diff model_

[Clyde Schechter]

When you want help interpreting results of models, it is far more helpful to show the actual model commands and outputs than to write an abstract equation. Please post back.

[Mathew Chandy]

Here is the code I used (NOTE: I tried using dataex to display results but it was not working so I had to copy and paste from stata directly.)

xtreg totalexportsA i.y2011##i.Country gdppc natresrent PartnershipAg2008,fe vce(robust)


note: 1.Country omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =         56
Group variable: c_id                            Number of groups  =          2

R-sq:                                           Obs per group:
     within  = 0.9397                                         min =         28
     between = 1.0000                                         avg =       28.0
     overall = 0.8147                                         max =         28

                                                F(1,1)            =          .
corr(u_i, Xb)  = 0.0526                         Prob > F          =          .

                                        (Std. Err. adjusted for 2 clusters in c_id)
-----------------------------------------------------------------------------------
                  |               Robust
    totalexportsA |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
          1.y2011 |  -225539.5   219203.1    -1.03   0.491     -3010779     2559700
        1.Country |          0  (omitted)
                  |
    y2011#Country |
             1 1  |   289602.3   55257.28     5.24   0.120    -412508.1    991712.6
                  |
            gdppc |   34.24794   5.770514     5.93   0.106    -39.07339    107.5693
       natresrent |  -6898.862   19706.81    -0.35   0.786    -257297.7    243499.9
PartnershipAg2008 |   231725.9   107552.2     2.15   0.277     -1134854     1598306
            _cons |  -199992.4   134639.5    -1.49   0.377     -1910749     1510764
------------------+----------------------------------------------------------------
          sigma_u |  311952.65
          sigma_e |  142935.56
              rho |  .82648461   (fraction of variance due to u_i)
-----------------------------------------------------------------------------------

xtreg totalexportsA i.y2011##i.Country i.Year gdppc natresrent PartnershipAg2008,fe vce(robust)

note: 1.Country omitted because of collinearity
note: 2017.Year omitted because of collinearity
note: PartnershipAg2008 omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =         56
Group variable: c_id                            Number of groups  =          2

R-sq:                                           Obs per group:
     within  = 0.9908                                         min =         28
     between = 1.0000                                         avg =       28.0
     overall = 0.9143                                         max =         28

                                                F(2,1)            =          .
corr(u_i, Xb)  = 0.1448                         Prob > F          =          .

                                        (Std. Err. adjusted for 2 clusters in c_id)
-----------------------------------------------------------------------------------
                  |               Robust
    totalexportsA |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
          1.y2011 |   396884.6   34273.79    11.58   0.055    -38605.18    832374.5
        1.Country |          0  (omitted)
                         |
y2011#Country
             1 1    |  -53157.68   .0046515 -1.1e+07   0.000    -53157.74   -53157.62
                     |
             Year |
            1991  |   -99988.7   79483.77    -1.26   0.428     -1109926    909948.4
            1992  |  -78935.72   42202.63    -1.87   0.313      -615171    457299.6
            1993  |   -82976.9   41967.42    -1.98   0.298    -616223.5    450269.7
            1994  |   -37952.1   20371.17    -1.86   0.314    -296792.3    220888.1
            1995  |  -32736.04   46618.73    -0.70   0.610    -625083.2    559611.1
            1996  |  -23275.03   72351.58    -0.32   0.802      -942589    896038.9
            1997  |   -56538.1   74294.02    -0.76   0.586     -1000533    887456.9
            1998  |  -119671.6    34068.8    -3.51   0.177    -552556.8    313213.5
            1999  |  -98245.19   38291.25    -2.57   0.237    -584781.7    388291.3
            2000  |   33288.03   16530.67     2.01   0.293      -176754      243330
            2001  |   37248.95   63668.13     0.59   0.663    -771731.4    846229.3
            2002  |   33272.89   22355.52     1.49   0.377    -250780.9    317326.7
            2003  |   26578.72   70829.26     0.38   0.771    -873392.3    926549.7
            2004  |   39336.36   137303.5     0.29   0.822     -1705270     1783943
            2005  |   157335.8   72720.86     2.16   0.276    -766670.3     1081342
            2006  |   296406.2   75431.05     3.93   0.159    -662036.1     1254849
            2007  |   359755.7   45575.52     7.89   0.080    -219336.1    938847.5
            2008  |   665752.4   75332.46     8.84   0.072    -291437.3     1622942
            2009  |   329512.6   7259.301    45.39   0.014     237274.4    421750.7
            2010  |   671341.8     111644     6.01   0.105    -747229.7     2089913
            2011  |   564353.4   11125.31    50.73   0.013       422993    705713.8
            2012  |   294781.1   230272.5     1.28   0.422     -2631109     3220671
            2013  |   230987.4   123773.2     1.87   0.313     -1341701     1803676
            2014  |   143661.6    15670.8     9.17   0.069    -55454.79    342778.1
            2015  |  -200498.5   126267.6    -1.59   0.358     -1804880     1403883
            2016  |  -148748.1   124794.7    -1.19   0.444     -1734415     1436918
            2017  |          0  (omitted)
                  |
            gdppc |    24.6533          .        .       .            .           .
       natresrent |  -48420.56          .        .       .            .           .
PartnershipAg2008 |          0  (omitted)
            _cons |   118161.6   12686.57     9.31   0.068    -43036.59    279359.7
------------------+----------------------------------------------------------------
          sigma_u |   239137.7
          sigma_e |  79579.277
              rho |  .90030083   (fraction of variance due to u_i)
-----------------------------------------------------------------------------------

[Clyde Schechter]

Your first model (without the i.year terms) is valid, and the second one is not properly specified.

Notice that in the second model, in addition to the usual first year being omitted, 2017 is also omitted. That's because the i.year variables are colinear with the y2011 variable. So the y2011 variable is no longer interpretable in the same way as it is in the first model where it is not colinear with anything. Consequently, y2011##Country is no longer a proper specification of a difference in difference model. What you can do, if you want to include year effects, is go to a generalized difference in difference model. But for that you would specify only y2011#Country (note the single #, not the double ##) along with your fixed effects and your i.year variables (along with the other covariates).

Now, there is no guarantee that the generalized DID model will agree with the DID model you present first, either. They are different models by virtue of the generalized one including year-specific shocks to the outcome. So you need to decide whether you think that year-specific shocks are relevant here or not.

A couple of other points not raised directly by your question but that I notice from your output:

1. With only two groups, robust standard errors are not valid and you should not use them.
2. In the model with year effects, PartnershipAg2008 is suddenly omitted. That may not be a problem, but be sure you understand why that is happening and whether it is OK, or whether it reflects errors in your data.

[Mathew Chandy]

Here are the results given your comments:

xtreg totalexportsA i.y2011#i.Country i.Year gdppc natresrent PartnershipAg2008,fe

note: 1.y2011#0b.Country omitted because of collinearity
note: 1.y2011#1.Country omitted because of collinearity
note: PartnershipAg2008 omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =         56
Group variable: c_id                            Number of groups  =          2

R-sq:                                           Obs per group:
     within  = 0.9908                                         min =         28
     between = 1.0000                                         avg =       28.0
     overall = 0.9378                                         max =         28

                                                F(30,24)          =      86.62
corr(u_i, Xb)  = 0.1917                         Prob > F          =     0.0000

-----------------------------------------------------------------------------------
    totalexportsA |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
    y2011#Country |
             0 1  |   53157.68   92772.57     0.57   0.572    -138315.5    244630.8
             1 0  |          0  (omitted)
             1 1  |          0  (omitted)
                  |
             Year |
            1991  |   -99988.7   80095.52    -1.25   0.224    -265297.7    65320.32
            1992  |  -78935.72   80257.64    -0.98   0.335    -244579.4    86707.91
            1993  |   -82976.9   80040.47    -1.04   0.310    -248172.3    82218.51
            1994  |   -37952.1   80188.89    -0.47   0.640    -203453.8    127549.6
            1995  |  -32736.04   81681.61    -0.40   0.692    -201318.6    135846.5
            1996  |  -23275.03   82692.78    -0.28   0.781    -193944.5    147394.5
            1997  |   -56538.1   85761.92    -0.66   0.516      -233542    120465.8
            1998  |  -119671.6   85846.09    -1.39   0.176    -296849.3    57505.99
            1999  |  -98245.19   85128.37    -1.15   0.260    -273941.5    77451.13
            2000  |   33288.03   83701.78     0.40   0.694      -139464      206040
            2001  |   37248.95   82130.59     0.45   0.654    -132260.3    206758.2
            2002  |   33272.89   84150.86     0.40   0.696    -140405.9    206951.7
            2003  |   26578.72   90756.77     0.29   0.772      -160734    213891.5
            2004  |   39336.36   104567.5     0.38   0.710    -176480.3    255153.1
            2005  |   157335.8     109836     1.43   0.165     -69354.5    384026.1
            2006  |   296406.2   115048.3     2.58   0.017     58958.28    533854.2
            2007  |   359755.7   130164.6     2.76   0.011     91109.25    628402.2
            2008  |   665752.4   137080.6     4.86   0.000     382831.9    948672.9
            2009  |   329512.6   116281.1     2.83   0.009     89520.27    569504.9
            2010  |   671341.8   129079.9     5.20   0.000     404933.9    937749.7
            2011  |     961238   185333.1     5.19   0.000     578729.3     1343747
            2012  |   691665.7   198447.3     3.49   0.002     282090.6     1101241
            2013  |     627872   200554.1     3.13   0.005     213948.7     1041795
            2014  |   540546.3   198097.8     2.73   0.012     131692.5      949400
            2015  |   196386.2   186064.5     1.06   0.302      -187632    580404.3
            2016  |   248136.5   166138.9     1.49   0.148    -94757.38    591030.4
            2017  |   396884.6   170754.7     2.32   0.029      44464.3      749305
                  |
            gdppc |    24.6533   4.078881     6.04   0.000     16.23491     33.0717
       natresrent |  -48420.56   10377.07    -4.67   0.000    -69837.79   -27003.33
PartnershipAg2008 |          0  (omitted)
            _cons |   91582.74   68836.22     1.33   0.196    -50488.23    233653.7
------------------+----------------------------------------------------------------
          sigma_u |  201549.54
          sigma_e |  79579.277
              rho |  .86512937   (fraction of variance due to u_i)
-----------------------------------------------------------------------------------
F test that all u_i=0: F(1, 24) = 10.61                      Prob > F = 0.0033

Given these results and the addition of year and country fixed effects, would it therefore show that the policy shock has no significant impact on the level of exports and that exports rose by $53,157.68? Or would this interpretation need to be expanded further?

[Clyde Schechter]

Well, something weird happened. It picked 0.y2011#1.Country to represent the interaction. I'm, not sure why that happened, but you should re-run it with 1.y2011#1.Country instead of i.y2011#i.Country, to be sure that the category of post-policy and intervention country is the one that gets represented. I think that will end up reversing the sign and you will be back to -53157 as your estimate of the policy effect.

However, it turns out, when reporting the effect, don't forget to report the confidence interval as well: it is very wide and what you are really getting here is an inconclusive result.

[Mathew Chandy]

Hi Clyde, you were right - the signs changed as you expected. One further question I had was that I'm not entirely sure why adding year fixed effects completely changes the result compared to simply using country fixed effects only - what does the inclusion of year fixed effects actually do that country fixed effects do not account for?

xtreg totalexportsA 1.y2011#1.Country i.Year gdppc natresrent PartnershipAg2008,fe
note: PartnershipAg2008 omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =         56
Group variable: c_id                            Number of groups  =          2

R-sq:                                           Obs per group:
     within  = 0.9908                                         min =         28
     between = 1.0000                                         avg =       28.0
     overall = 0.9143                                         max =         28

                                                F(30,24)          =      86.62
corr(u_i, Xb)  = 0.1448                         Prob > F          =     0.0000

-----------------------------------------------------------------------------------
    totalexportsA |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
    y2011#Country |
             1 1  |  -53157.68   92772.57    -0.57   0.572    -244630.8    138315.5
                  |
             Year |
            1991  |   -99988.7   80095.52    -1.25   0.224    -265297.7    65320.32
            1992  |  -78935.72   80257.64    -0.98   0.335    -244579.4    86707.91
            1993  |   -82976.9   80040.47    -1.04   0.310    -248172.3    82218.51
            1994  |   -37952.1   80188.89    -0.47   0.640    -203453.8    127549.6
            1995  |  -32736.04   81681.61    -0.40   0.692    -201318.6    135846.5
            1996  |  -23275.03   82692.78    -0.28   0.781    -193944.5    147394.5
            1997  |   -56538.1   85761.92    -0.66   0.516      -233542    120465.8
            1998  |  -119671.6   85846.09    -1.39   0.176    -296849.3    57505.99
            1999  |  -98245.19   85128.37    -1.15   0.260    -273941.5    77451.13
            2000  |   33288.03   83701.78     0.40   0.694      -139464      206040
            2001  |   37248.95   82130.59     0.45   0.654    -132260.3    206758.2
            2002  |   33272.89   84150.86     0.40   0.696    -140405.9    206951.7
            2003  |   26578.72   90756.77     0.29   0.772      -160734    213891.5
            2004  |   39336.36   104567.5     0.38   0.710    -176480.3    255153.1
            2005  |   157335.8     109836     1.43   0.165     -69354.5    384026.1
            2006  |   296406.2   115048.3     2.58   0.017     58958.28    533854.2
            2007  |   359755.7   130164.6     2.76   0.011     91109.25    628402.2
            2008  |   665752.4   137080.6     4.86   0.000     382831.9    948672.9
            2009  |   329512.6   116281.1     2.83   0.009     89520.27    569504.9
            2010  |   671341.8   129079.9     5.20   0.000     404933.9    937749.7
            2011  |     961238   185333.1     5.19   0.000     578729.3     1343747
            2012  |   691665.7   198447.3     3.49   0.002     282090.6     1101241
            2013  |     627872   200554.1     3.13   0.005     213948.7     1041795
            2014  |   540546.3   198097.8     2.73   0.012     131692.5      949400
            2015  |   196386.2   186064.5     1.06   0.302      -187632    580404.3
            2016  |   248136.5   166138.9     1.49   0.148    -94757.38    591030.4
            2017  |   396884.6   170754.7     2.32   0.029      44464.3      749305
                  |
            gdppc |    24.6533   4.078881     6.04   0.000     16.23491     33.0717
       natresrent |  -48420.56   10377.07    -4.67   0.000    -69837.79   -27003.33
PartnershipAg2008 |          0  (omitted)
            _cons |   118161.6   89774.12     1.32   0.201     -67123.1    303446.2
------------------+----------------------------------------------------------------
          sigma_u |   239137.7
          sigma_e |  79579.277
              rho |  .90030083   (fraction of variance due to u_i)
-----------------------------------------------------------------------------------
F test that all u_i=0: F(1, 24) = 44.75                      Prob > F = 0.0000

[Clyde Schechter]

In the model without i.year, you are fitting a model in which everything other than the intervention and pre-post 2011, gdppc, natresrent, and PartnershipAg2008 is just random noise. When you include i.year you are saying that in addition to those explanatory factors there are yearly shocks to the outcome variable which are not to be considered noise but are to be explicitly estimated. Since the other explanatory variables are not constant within years (except, it appears, PartnershipAg2008), explanatory variance gets reallocated among the variables, with the yearly shocks getting to do some of the heavy lifting. In your situation, the yearly shocks are huge. They are, if you will, "sucking up all the oxygen." Also, in your particular results there is a tendency for the earlier years to have negative shocks and the later years to have positive shocks that increase over time: there is something of an upward trend in your outcome variable here which may well be "drowning out" the effect of the intervention. Notice in particular that your shock for 2011 is 961,238, the largest shock of them all--precisely the year your intervention kicks in! That yearly shock is overwhelming the intervention effect.

Finally, with a grand total of 56 cases, you really have no hope of getting anything meaningful out of a regression with 30 predictor variables. The ratio of observations to variables here isn't even 2! With only 56 cases you really are on thin ice even just modeling two predictors. I think you are trying to squeeze blood from a stone here.

[Susan Jain]

Why do my coefficients change when I incorporate country fixed and year fixed effect vis-a-vis only year fixed effect? how can I explain this reasoning? I am studying FDI determinants in developed nations.
pl guide.
many thanks in anticipation

[Clyde Schechter]

Why wouldn't they change. Any time you add or remove variables to or from a model, everything can change. Sometimes the changes are large, sometimes they are small. But the absence of change can occur only if the variable added or removed is independent of the other variables in the model, which rarely happens in real life.

In your particular situation, the addition of country fixed effects has resulted in changes. This is to be expected if different countries have different outcomes and have different values of the explanatory variables. It would be astonishing if that were not the case!

[Jared Greathouse]

Susan Jain what are unit/time fixed effects anyways? Why do we include them in econonetric modeling?

[Susan Jain]

@Clyde Schechter - May I request Prof Clyde Schechter to pl help me in understanding -Why do my coefficients change when I incorporate country fixed and year fixed effect vis-a-vis only year fixed effect? how can I explain this reasoning? I am studying FDI determinants in developed nations.
pl guide.
many thanks in anticipation

[Jared Greathouse]

Susan Jain So you're just going to skip over my question as though I said nothing? Clyde explained it already quite well, so I'm explaining it slightly differently. All I'm asking you is what fixed effects are and why we include them.

[Clyde Schechter]

Why do my coefficients change when I incorporate country fixed and year fixed effect vis-a-vis only year fixed effect? how can I explain this reasoning?

When you incorporate country and year fixed effects, that is the same thing as adding country fixed effects to a year-fixed-effects only model. As already explained, the change in results requires no explanation: it is entirely expected. Whenever you change a model, you should expect that the results would change. It's a different set of equations being solved.

If the results didn't change after you change the model you would definitely have to come up with an explanation for that.

[Susan Jain]

@Clyde Schechter Thank you so much Prof.
Much appreciate.