Adding controls, insignificant treatment var turns significant

[Sophie Ren]

Dear friends,

I am running a Fixed Effects regression to test the effect of a policy on firms' patent application. The outcome variable is number of patents.

I first run the simple model with FEs but without any controls:

Conditional fixed-effects Poisson regression    Number of obs     =     36,188
Group variable: firm_id                         Number of groups  =      5,854

                                                Obs per group:
                                                              min =          2
                                                              avg =        6.2
                                                              max =          9

                                                Wald chi2(9)      =     541.81
Log pseudolikelihood  = -28813.085              Prob > chi2       =     0.0000

                                    (Std. Err. adjusted for clustering on firm_id)
----------------------------------------------------------------------------------
                 |               Robust
 application_num |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
         treated |   .3129292   .4317598     0.72   0.469    -.5333044    1.159163
                 |
application_year |
           1999  |    .618862   .1711453     3.62   0.000     .2834233    .9543006
           2000  |     .92752   .3302112     2.81   0.005     .2803178    1.574722
           2001  |   1.252948   .4390228     2.85   0.004     .3924788    2.113416
           2002  |   2.116918   .4108072     5.15   0.000     1.311751    2.922085
           2003  |   2.576106   .4007325     6.43   0.000     1.790685    3.361527
           2004  |   2.843799    .434273     6.55   0.000     1.992639    3.694958
           2005  |   3.167553   .4819282     6.57   0.000     2.222991    4.112115
           2006  |   3.545173   .5469874     6.48   0.000     2.473097    4.617249
----------------------------------------------------------------------------------

Now the Treated variable is statistically insignificant.
I then add controls:

Conditional fixed-effects Poisson regression    Number of obs     =     36,188
Group variable: firm_id                         Number of groups  =      5,854

                                                Obs per group:
                                                              min =          2
                                                              avg =        6.2
                                                              max =          9

                                                Wald chi2(11)     =    1271.57
Log pseudolikelihood  = -28331.198              Prob > chi2       =     0.0000

                                        (Std. Err. adjusted for clustering on firm_id)
--------------------------------------------------------------------------------------
                     |               Robust
     application_num |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
---------------------+----------------------------------------------------------------
             treated |   .6178822   .2446853     2.53   0.012     .1383078    1.097457
cum_app_agg_last_log |   1.126586   .4889455     2.30   0.021     .1682705    2.084902
   import_export_log |  -.8108906    .644505    -1.26   0.208    -2.074097    .4523159
                     |
    application_year |
               1999  |   .5374376   .2319735     2.32   0.021     .0827778    .9920974
               2000  |   .7382168   .4147308     1.78   0.075    -.0746406    1.551074
               2001  |   .7904961   .6667538     1.19   0.236    -.5163174     2.09731
               2002  |   1.539174   .9464433     1.63   0.104    -.3158206    3.394169
               2003  |   1.931999   1.232849     1.57   0.117      -.48434    4.348339
               2004  |   2.068571   1.526228     1.36   0.175      -.92278    5.059923
               2005  |   2.223375   1.689623     1.32   0.188    -1.088226    5.534975
               2006  |   2.407712   1.887254     1.28   0.202    -1.291238    6.106662
--------------------------------------------------------------------------------------

As you can see, after adding controls, the Treated variable turns statistically significant. cum_app_agg_last_log is the cumulative number of patents in a state up to the year before policy launch.

Can anyone help me understand why this is the case? I know that often time adding controls turns a significant variable insignificant as the controls can absorb some explanation power. But I just can't figure out why in my case, a insignificant variable becomes significant once controls are added.

And which result shall I trust? Does the policy really have a significant impact on firms' patent application?

Thank you very much!

[Dimitriy V. Masterov]

Relevant controls can absorb some of the noise, making the treatment easier to detect.

But your effect also flips sign when you add controls. It's not just a change in significance.

[Sophie Ren]

Hi Dimitriy,

Thank you for your reply! Sorry I edited my post a minute after you saw it. I copied the wrong table originally.

But I see your point! Thanks a lot!

[Sophie Ren]

Um... I hope this is not too much... But can anyone help me understand this in a statistical sense?

Sorry my econometrics is really weak... I just can't think out the mechanisms...

Thanks a million!

[Phil Bromiley]

At least two things are happening. First, the parameter on treated is estimated holding all the other variables constant. When you hold different variables constant, the effect of treated changes. One way of thinking of this is that when you have multiple rhs variables, only the variance that each variable has that is not shared with the other rhs variables can be used to determine the parameter. Changing the rhs variables changes what is not shared variance.
So, if I looked at exercise and health, I might get a very different result if I just run
reg health exercise
then if I ran
reg health exercise age.

The second thing that can happen is the additional controls change the error variance so even if they don't change the parameter value, they can change the standard error/significance.