Different Result of F-statistics with Year Fixed Effect with Reghdfe

[Reza Mahardika]

Dear All,

I am using reghdfe in Stata/MP 18.0 for Windows

I find that the F-statistics result of using reghdfe by includes year fixed effect is different: whether the year fixed effect is in absorb(year) or using i.year. Furthermore, the result is like below:

The first code is:

reghdfe ln_allocdak_f_realpc lead* lag* , absorb(year code_district ) cluster (code_district)

The result is shown below

.         reghdfe ln_allocdak_f_realpc lead* lag* , absorb(year code_district ) cluster (code_d
> istrict) 
(MWFE estimator converged in 3 iterations)

HDFE Linear regression                            Number of obs   =      3,719
Absorbing 2 HDFE groups                           F(   8,    464) =       1.78
Statistics robust to heteroskedasticity           Prob > F        =     0.0791
                                                  R-squared       =     0.8989
                                                  Adj R-squared   =     0.8839
                                                  Within R-sq.    =     0.0033
Number of clusters (code_district) =        465   Root MSE        =     0.3588

                        (Std. err. adjusted for 465 clusters in code_district)
------------------------------------------------------------------------------
             |               Robust
ln_allocda.. | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       lead6 |  -.0742988   .0767775    -0.97   0.334    -.2251734    .0765758
       lead5 |  -.1002508   .0496612    -2.02   0.044    -.1978395   -.0026621
       lead4 |  -.0188273     .04925    -0.38   0.702     -.115608    .0779535
       lead3 |  -.0422806   .0422879    -1.00   0.318    -.1253802    .0408189
       lead2 |   -.049646    .040094    -1.24   0.216    -.1284344    .0291424
        lag0 |  -.0745906   .0513634    -1.45   0.147    -.1755243     .026343
        lag1 |  -.1492528   .0600643    -2.48   0.013    -.2672847    -.031221
        lag2 |   -.054058   .0626161    -0.86   0.388    -.1771042    .0689882
       _cons |   5.078007   .0070776   717.48   0.000     5.064099    5.091915
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-------------------------------------------------------+
   Absorbed FE | Categories  - Redundant  = Num. Coefs |
---------------+---------------------------------------|
          year |         8           1           7     |
 code_district |       465         465           0    *|
-------------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

The second code is using this

reghdfe ln_allocdak_f_realpc lead* lag* i.year, absorb(code_district) cluster (code_district)

reghdfe ln_allocdak_f_realpc lead* lag* i.year, absorb(code_district) cluster (code_d
> istrict)
(MWFE estimator converged in 1 iterations)

HDFE Linear regression                            Number of obs   =      3,719
Absorbing 1 HDFE group                            F(  15,    464) =      91.98
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.8989
                                                  Adj R-squared   =     0.8839
                                                  Within R-sq.    =     0.3754
Number of clusters (code_district) =        465   Root MSE        =     0.3588

                        (Std. err. adjusted for 465 clusters in code_district)
------------------------------------------------------------------------------
             |               Robust
ln_allocda.. | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       lead6 |  -.0742988   .0767775    -0.97   0.334    -.2251734    .0765758
       lead5 |  -.1002508   .0496612    -2.02   0.044    -.1978395   -.0026621
       lead4 |  -.0188273     .04925    -0.38   0.702     -.115608    .0779535
       lead3 |  -.0422806   .0422879    -1.00   0.318    -.1253802    .0408189
       lead2 |   -.049646    .040094    -1.24   0.216    -.1284344    .0291424
        lag0 |  -.0745906   .0513634    -1.45   0.147    -.1755243     .026343
        lag1 |  -.1492528   .0600643    -2.48   0.013    -.2672847    -.031221
        lag2 |   -.054058   .0626161    -0.86   0.388    -.1771042    .0689882
             |
        year |
       2018  |  -.1944722   .0189415   -10.27   0.000    -.2316939   -.1572506
       2019  |  -.1225608   .0200641    -6.11   0.000    -.1619886   -.0831331
       2020  |  -.4764936   .0230499   -20.67   0.000    -.5217886   -.4311985
       2021  |  -.3060795   .0236001   -12.97   0.000    -.3524557   -.2597032
       2022  |  -.4390548   .0269705   -16.28   0.000    -.4920543   -.3860552
       2023  |  -.7413073   .0335241   -22.11   0.000    -.8071851   -.6754295
       2024  |   -.737513   .0330956   -22.28   0.000    -.8025489   -.6724771
             |
       _cons |   5.455241   .0189759   287.48   0.000     5.417952     5.49253
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-------------------------------------------------------+
   Absorbed FE | Categories  - Redundant  = Num. Coefs |
---------------+---------------------------------------|
 code_district |       465         465           0    *|
-------------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

Any idea what is happening? Why the f-statistics and p-value for F different? Thank you very much

[Carlo Lazzaro]

Reza:
it is a matter of degrees of freedom:
1) in your first -reghdfe- specification you have 8 predictors+_cons and 464 observations;
2) in your second one you have 15+_cons and 464 observations.

[Clyde Schechter]

The F-test for the model is the test of the joint hypothesis that the coefficients of all of the non-absorbed variables are zero. So that F-test is testing different null hypotheses in those two models. In the model with i.year, the coefficients of the year indicators being zero is part of the null hypothesis, but in the model that absorbs year, they are not. You can also see this reflected in the different numerator degrees of freedom for the F test. There are 7 more numerator degrees of freedom in the i.year model, corresponding to the 7 year indicators.

Other than intellectual curiosity and wanting to understand what your analysis is doing (which I commend), why do you care about this? The overall model F test is almost never of any importance, because if any null hypothesis were ever a straw man (and I would argue that nearly all of them are), the overall model F-test surely almost always is one.

Added: Crossed with #2.

[George Ford]

I find the F-test useful for some things. In your case, your x's don't have much juice, which is somewhat apparent from the t-stats but not obvious. Your model has some, but low, predictive capability. Getting significance of the F from yearly fixed effects is clever but, in my opinion, is a form of p-hacking the F.

[Reza Mahardika]

Carlo Lazzaro Thank you very much for your explanation!
Clyde Schechter Thank you for you answer. I just want to check what happened between those two regressions. If something different, perhaps I am not fully understand the command and I want to learn more about that. Thank you again
George Ford Yes, I am agree with your notion about "p-hacking". I'll check the model again. Thank you!