Significant after adding controls but not significant otherwise

[Meghaa --]

Hi,

I am looking to see the impact of Same Sex legalisation in USA states on hate crime against LGBT numbers. (Using states as cross sectional units)
I ran a FIxed effect regression and a Two way fixed effect regression with i.Year.

I ran another regression with a bunch of controls.

Based on the results I see that with Fixed effects and TWFE my coeffecient is isignificant but closer to significance at 0.063 with fixed effect.
The coffecient sign is different in both models.
But when I add controls my p value drops and Fixed effect model with controls becomes significant and p value for TWFE with controls in 0.051.

My chosen methodology is TWFE.
My concerns are:
Why does it become relatively more significant after controls and why the coffecient sign is different for the two models.
I would really appreciate if somebody can help me interpret why this is, results seems odd.

1. xtreg Sexualorientation_Bias treated, fe

Fixed-effects (within) regression Number of obs = 1,377
Group variable: State_nume~c Number of groups = 51

R-squared: Obs per group:
Within = 0.0026 min = 27
Between = 0.0267 avg = 27.0
Overall = 0.0001 max = 27

F(1,1325) = 3.45
corr(u_i, Xb) = -0.0284 Prob > F = 0.0634

------------------------------------------------------------------------------
Sexualorie~s | Coefficient Std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treated | -1.710473 .920559 -1.86 0.063 -3.516385 .0954393
_cons | 23.97606 .5163818 46.43 0.000 22.96304 24.98907
-------------+----------------------------------------------------------------
sigma_u | 40.268787
sigma_e | 15.714549
rho | .86783832 (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(50, 1325) = 177.15 Prob > F = 0.0000


2. xtreg Sexualorientation_Bias treated i.Year, fe

Fixed-effects (within) regression Number of obs = 1,377
Group variable: State_nume~c Number of groups = 51

R-squared: Obs per group:
Within = 0.0184 min = 27
Between = 0.0267 avg = 27.0
Overall = 0.0026 max = 27

F(27,1299) = 0.90
corr(u_i, Xb) = 0.0017 Prob > F = 0.6068

------------------------------------------------------------------------------
Sexualorie~s | Coefficient Std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treated | .2751672 2.444596 0.11 0.910 -4.520621 5.070956
|
Year |
1996 | -.0784314 3.117859 -0.03 0.980 -6.195023 6.03816
1997 | 1.784314 3.117859 0.57 0.567 -4.332278 7.900905
1998 | 5.352941 3.117859 1.72 0.086 -.7636502 11.46953
1999 | 5.941176 3.117859 1.91 0.057 -.1754149 12.05777
2000 | 6.117647 3.117859 1.96 0.050 .0010557 12.23424
2001 | 7.529412 3.117859 2.41 0.016 1.41282 13.646
2002 | 2.490196 3.117859 0.80 0.425 -3.626395 8.606787
2003 | 4.529412 3.117859 1.45 0.147 -1.58718 10.646
2004 | 3.347546 3.118228 1.07 0.283 -2.769768 9.46486
2005 | 1.112252 3.118228 0.36 0.721 -5.005063 7.229566
2006 | 3.504408 3.118228 1.12 0.261 -2.612906 9.621723
2007 | 5.053428 3.118228 1.62 0.105 -1.063886 11.17074
2008 | 6.204895 3.119333 1.99 0.047 .0854133 12.32438
2009 | 4.05685 3.123749 1.30 0.194 -2.071296 10.185
2010 | 4.869588 3.131096 1.56 0.120 -1.27297 11.01215
2011 | 5.589683 3.135862 1.78 0.075 -.5622255 11.74159
2012 | 5.402421 3.147563 1.72 0.086 -.7724421 11.57729
2013 | 4.712199 3.222585 1.46 0.144 -1.609842 11.03424
2014 | .2567447 3.563528 0.07 0.943 -6.734156 7.247645
2015 | .6464014 3.961956 0.16 0.870 -7.126132 8.418935
2016 | 1.469931 3.961956 0.37 0.711 -6.302603 9.242464
2017 | 2.156205 3.961956 0.54 0.586 -5.616328 9.928739
2018 | 3.332676 3.961956 0.84 0.400 -4.439858 11.10521
2019 | 2.822872 3.961956 0.71 0.476 -4.949662 10.59541
2020 | 1.097382 3.961956 0.28 0.782 -6.675152 8.869915
2021 | 1.371892 3.961956 0.35 0.729 -6.400642 9.144425
|
_cons | 19.98039 2.20466 9.06 0.000 15.65531 24.30548
-------------+----------------------------------------------------------------
sigma_u | 40.242482
sigma_e | 15.744419
rho | .86725172 (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(50, 1299) = 175.58 Prob > F = 0.0000

3. xtreg Sexualorientation_Bias treated Poverty Unemployment TotalHATECRIME TotalCrimeviolentproperty demRepgov, fe

Fixed-effects (within) regression Number of obs = 1,377
Group variable: State_nume~c Number of groups = 51

R-squared: Obs per group:
Within = 0.6659 min = 27
Between = 0.9246 avg = 27.0
Overall = 0.8904 max = 27

F(6,1320) = 438.50
corr(u_i, Xb) = 0.2255 Prob > F = 0.0000

----------------------------------------------------------------------------------------
Sexualorientation_Bias | Coefficient Std. err. t P>|t| [95% conf. interval]
-----------------------+----------------------------------------------------------------
treated | -1.518882 .6123746 -2.48 0.013 -2.720215 -.3175479
Poverty | .0000127 1.90e-06 6.69 0.000 8.98e-06 .0000164
Unemployment | -5.00e-06 3.23e-06 -1.55 0.122 -.0000113 1.34e-06
TotalHATECRIME | .1497081 .0030646 48.85 0.000 .1436961 .1557201
TotalCrimeviolentpro~y | -.0000284 5.68e-06 -4.99 0.000 -.0000395 -.0000172
demRepgov | 1.715274 .5932629 2.89 0.004 .5514329 2.879115
_cons | -1.555793 2.045442 -0.76 0.447 -5.568466 2.45688
-----------------------+----------------------------------------------------------------
sigma_u | 11.383083
sigma_e | 9.1121406
rho | .60945962 (fraction of variance due to u_i)
----------------------------------------------------------------------------------------
F test that all u_i=0: F(50, 1320) = 38.36 Prob > F = 0.0000

[Carlo Lazzaro]

Meghaa (please see among the FAQ the preference for full names on this forum. Thanks),
welcome on the Statalist.
The first advice, with 51 panels, is to use cluster robust standard errors (see -robust- or -vce(cluster clusterid)- otions available from -xtreg-).

[George Ford]

Absent the year fixed effect, you risk spurious correlation over time. The law likely appear later in the sample and the dependent variable may be trending as well. Use TWFE.

You could use -reghdfe- as an alternative to xtreg, and put the year FE in the absorb. And cluster(states).as Carlo suggests.

reghdfe Sexualorientation_Bias treated Poverty Unemployment TotalHATECRIME TotalCrimeviolentproperty demRepgov, absorb(state year) cluster(state)

The richer model is probably better.

Is there a period in the sample when all states have no law? If so, you've got a difference-in-difference model, and you should exploit that. As the laws are arriving at different times, csdid would work, or eventdd.

Does treated == 1 after the law is passed, or over the entire sample? If after, then again you've got a DID model, but you need to deal with any state that is treated==1 for all years. Normally, these are dropped, but there are options to use them.

If states are either treated or untreated for the entire time, you're looking at a means difference model. You might considering a matching algorithm to address confounding.

Also, I suspect someone will argue that the passing of a law may be endogenous, which opens a can of worms of a different sort. I suspect political leaning variables might be good instruments.

[Meghaa --]

Dear Carlo,

I am aware of preference for full name but legally I do have just the first name.
And thank you for suggesting clustering, I tried that as well. with clustering the p value at 95% changes but again even though not significant the p value is smaller with controls.

xtreg Sexualorientation_Bias treated Poverty Unemployment TotalHATECRIME TotalCrimeviolentprope
> rty demRepgov i.Year, fe cluster(State)

Fixed-effects (within) regression Number of obs = 1377
Group variable: state_id Number of groups = 51

R-sq: Within = 0.6884 Obs per group: min = 27
Between = 0.9241 avg = 27.0
Overall = 0.8921 max = 27

F(32,50) = 90.90
corr(u_i, Xb) = 0.3335 Prob > F = 0.0000

(Std. err. adjusted for 51 clusters in State)
-------------------------------------------------------------------------------------------
| Robust
Sexualorientation_Bias | Coefficient std. err. t P>|t| [95% conf. interval]
--------------------------+----------------------------------------------------------------
treated | 2.718662 1.783901 1.52 0.134 -.8644087 6.301734
Poverty | .0000102 5.25e-06 1.93 0.059 -3.88e-07 .0000207
Unemployment | -6.42e-06 5.58e-06 -1.15 0.255 -.0000176 4.78e-06
TotalHATECRIME | .1505782 .021932 6.87 0.000 .1065265 .1946298
TotalCrimeviolentproperty | -.0000257 .0000115 -2.23 0.030 -.0000489 -2.58e-06
demRepgov | 1.482157 .8750878 1.69 0.097 -.275509 3.239822
|
Year |
1996 | -2.803729 1.50145 -1.87 0.068 -5.819481 .2120222
1997 | 1.055485 1.15776 0.91 0.366 -1.269944 3.380914
1998 | 5.059477 1.682568 3.01 0.004 1.67994 8.439014
1999 | 5.471906 2.11707 2.58 0.013 1.219646 9.724166
2000 | 4.920137 1.913857 2.57 0.013 1.076041 8.764233
2001 | 1.831578 1.786901 1.03 0.310 -1.757519 5.420675
2002 | 3.376408 1.715417 1.97 0.055 -.0691083 6.821923
2003 | 4.831628 1.821075 2.65 0.011 1.173891 8.489366
2004 | 2.84587 1.522484 1.87 0.067 -.21213 5.90387
2005 | 1.053213 1.463236 0.72 0.475 -1.885782 3.992208
2006 | 2.301219 1.796351 1.28 0.206 -1.306857 5.909296
2007 | 4.011386 1.930603 2.08 0.043 .1336558 7.889117
2008 | 3.857292 1.865222 2.07 0.044 .1108833 7.6037
2009 | 5.486893 1.911898 2.87 0.006 1.646733 9.327053
2010 | 5.4496 1.878362 2.90 0.006 1.676798 9.222402
2011 | 6.653222 2.041938 3.26 0.002 2.551869 10.75457
2012 | 5.216818 2.557144 2.04 0.047 .0806425 10.35299
2013 | 5.485169 2.262736 2.42 0.019 .9403305 10.03001
2014 | 1.118218 2.075976 0.54 0.593 -3.051503 5.287938
2015 | .1622512 2.368664 0.07 0.946 -4.595351 4.919854
2016 | .0736614 2.553416 0.03 0.977 -5.055025 5.202348
2017 | -2.093522 2.851012 -0.73 0.466 -7.819949 3.632905
2018 | -.2438977 2.621321 -0.09 0.926 -5.508976 5.021181
2019 | -1.09054 2.880233 -0.38 0.707 -6.875658 4.694578
2020 | -4.373068 3.41217 -1.28 0.206 -11.22661 2.480477
2021 | -1.965585 3.326151 -0.59 0.557 -8.646356 4.715186
|
_cons | -3.387465 5.345975 -0.63 0.529 -14.12517 7.350241
--------------------------+----------------------------------------------------------------
sigma_u | 11.869783
sigma_e | 8.8883185
rho | .64072587 (fraction of variance due to u_i)

xtreg Sexualorientation_Bias treated i.Year, fe cluster (State)

Fixed-effects (within) regression Number of obs = 1377
Group variable: state_id Number of groups = 51

R-sq: Within = 0.0184 Obs per group: min = 27
Between = 0.0267 avg = 27.0
Overall = 0.0026 max = 27

F(27,50) = 4.95
corr(u_i, Xb) = 0.0017 Prob > F = 0.0000

(Std. err. adjusted for 51 clusters in State)
------------------------------------------------------------------------------
| Robust
Sexualorie~s | Coefficient std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
treated | .2751672 2.306874 0.12 0.906 -4.358326 4.90866
|
Year |
1996 | -.0784314 1.586919 -0.05 0.961 -3.265851 3.108989
1997 | 1.784314 1.416856 1.26 0.214 -1.061525 4.630152
1998 | 5.352941 2.143251 2.50 0.016 1.048095 9.657788
1999 | 5.941176 2.964803 2.00 0.051 -.0138057 11.89616
2000 | 6.117647 2.796682 2.19 0.033 .5003455 11.73495
2001 | 7.529412 3.07721 2.45 0.018 1.348654 13.71017
2002 | 2.490196 2.389557 1.04 0.302 -2.30937 7.289762
2003 | 4.529412 1.825248 2.48 0.016 .863293 8.195531
2004 | 3.347546 2.674913 1.25 0.217 -2.025175 8.720266
2005 | 1.112252 2.25137 0.49 0.623 -3.409757 5.634261
2006 | 3.504408 2.59725 1.35 0.183 -1.712322 8.721139
2007 | 5.053428 2.620467 1.93 0.059 -.2099345 10.31679
2008 | 6.204895 2.263361 2.74 0.008 1.658802 10.75099
2009 | 4.05685 2.681107 1.51 0.137 -1.328313 9.442012
2010 | 4.869588 1.940324 2.51 0.015 .9723324 8.766844
2011 | 5.589683 2.440097 2.29 0.026 .6886036 10.49076
2012 | 5.402421 2.975155 1.82 0.075 -.5733533 11.3782
2013 | 4.712199 2.81713 1.67 0.101 -.9461734 10.37057
2014 | .2567447 2.989621 0.09 0.932 -5.748086 6.261576
2015 | .6464014 3.06768 0.21 0.834 -5.515215 6.808018
2016 | 1.469931 2.927447 0.50 0.618 -4.410019 7.34988
2017 | 2.156205 3.085196 0.70 0.488 -4.040594 8.353004
2018 | 3.332676 3.235098 1.03 0.308 -3.165209 9.830561
2019 | 2.822872 3.157989 0.89 0.376 -3.520135 9.165879
2020 | 1.097382 3.62685 0.30 0.763 -6.187361 8.382125
2021 | 1.371892 6.485584 0.21 0.833 -11.65479 14.39857
|
_cons | 19.98039 1.834012 10.89 0.000 16.29667 23.66411
-------------+----------------------------------------------------------------
sigma_u | 40.242482
sigma_e | 15.744419
rho | .86725172 (fraction of variance due to u_i)


Please let me know if you have any other suggestions.

[Meghaa --]

Hi George,

Thank you for replying, I did use regdfe commands and they gave similar results and I clustered them today. As I mentioned it to Carlo the isssue remains the same, p value is smaller when using controls.

I am confused when you say to use TWFE but drop year(?). I was going for TWFE with Year and State level fixed effects, if not year what other fixed effect do you suggest I control for?

Yes, Massachussetts was the first state to legalise in 2004, so all states before that had it banned. I can exploit that using DID, thank you.

treated == 1 after it is implemented yes, example- it was implemented in 2015 in Alabama by SC decision, so I have all years including and after 2015 as 1. And No, no state has treated==1 for all years.

Please let me know your thoughts.

[Carlo Lazzaro]

Meghaa:
thanks for clarifying.
As the within R-sq is the most relevant with -fe-, I would stick with your first code. Please note that statistical significance is a very poor yardstick to assess the correctness of a regression specification.
As an aside, please use CODE delimiters to share what you posted and what Stata gave you back (as per FAQ). Thanks..

[George Ford]

There are several considerations here.

First, I don't think you have the right model. You've got data from 1996 on, and the first gay marriage legalization was in 2008 and most of them around 2013-2015 (depending on how you measure legality; bill or judicial). This is more a Diff-in-Diff story.

Second, how you define the dependent variable matters. These are rare events, so you have a count variable. You can express the DV on a per-million population scale, or the ratio of lgbtq crimes to total hate crimes, etc...

Third, total hate crimes should not include the LGBTQ hate crimes. The variable should be other hate crimes.

Fourth, given your apparent experience, I would suggest using eventdd. This is an event-type approach that estimates how the dep variables changes after the treatment by centering all the treatment times. It has a graphical element that is easy to interpret.

help eventdd

You need to create a variable that centers on the legislation passing year (so it will be negative before the year and positive after the year). (datayear - treatyear)

I downloaded the hate crime data and population/income data.

The crime data have missing years for some states, and I assume that the hate crime values were 0 in those years (which may not be true).

eventdd was used.

Y1 is the ratio of LGBTQ hate crimes per one-million in population. For the Xs, I used O1 as other hate crimes per million and ln(income) and absorbed state and year fixed effects using the hdfe option.

Don't see much effect, but this model is sparse and I've done this hastily. Other formulations of the DV might be interesting (ratio of LGBTQ/TotalHate).

You could also try csdid. I didn't find much there, but there was a few significant differences (though no real pattern and largely restricted to early adopters).

[Meghaa --]

Dear Carlo and George

Thank you for all your help, I ran regressions as suggested.

Thank you for mentioning DID George, I will try that.