Comparing joint significance across subsamples

[Kostas Pappas]

I need some help figuring a proper test for doing the following:

I am running the same regression across two subsamples.

I want to compare the joint significance of set of independent variables from subsample 1 versus for subsample 2.

Is there any test to demonstrate that a set of independent variables, jointly, have a more significant impact in subsample 1 than in subsample 2?

For example:

Y = X1 + X2 +X3 + X4 for subsample 1

Y = X1 + X2 +X3 + X4 for subsample 2

Then test that jointly X2, X3, X4 from subsample 1 are more significant than X2, X3, X4 from subsample 2.

Any help would be appreciated.

[George Ford]

Aside from the question of whether that makes any sense,

use auto, clear
reg price headroom trunk weight if foreign
test headroom trunk weight
local p1 = r(p)
reg price headroom trunk weight if ~foreign
test headroom trunk weight
local p2 = r(p)
if `p1'<`p2' {
    di "Model 1 More Significant"
}
else {
    di "Model 2 More Significant"
}

[Leonardo Guizzetti]

The approach suggested in #2 is not generally (*) statistically valid.

The question posed in #1 is vague and not easily understood because knowing whether a set of variables is more or less significant doesn't mean anything.

The aim is to compare two groups to each other, this much is clear. One way to do this (in a regression framework) is to introduce an interaction term. A test of an interaction term is interpreted as a test of differences between groups about the variables interacted. An example follows, building on that from #2.

sysuse auto
reg price headroom if foreign==0, nohead
reg price headroom if foreign==1, nohead
reg price i.foreign##c.headroom, nohead
lincom _b[headroom]
lincom _b[headroom] + _b[1.foreign#c.headroom]
testparm i.foreign#c.headroom

This can be extended to a joint test over multiple interaction terms. The null hypothesis is still that there is no difference between groups, but the alternative hypothesis is that there is some difference, but it is not known exactly where this difference is. While statistically valid, this result may not be easily interpretable.

sysuse auto
reg price headroom turn trunk if foreign==0, nohead
reg price headroom turn trunk if foreign==1, nohead
reg price i.foreign##c.headroom i.foreign##c.turn i.foreign##c.trunk, nohead
lincom _b[headroom]
lincom _b[headroom] + _b[1.foreign#c.headroom]
lincom _b[turn]
lincom _b[turn] + _b[1.foreign#c.turn]
lincom _b[trunk]
lincom _b[trunk] + _b[1.foreign#c.trunk]
testparm i.foreign#c.headroom i.foreign#c.turn i.foreign#c.trunk  // joint test

(*) In some simple cases, this is true. For example, with a test of effect size when n is fixed, then the p-value is a monotonic function of the t-statistic. Thus, a direct comparison of p-values implies a direct comparison of effect size. In a related direction, I recommend reading the article by Gelman and Stern, The Difference between "Significant" and "Not Significant" Is Not Itself Statistically Significant.

[Carlo Lazzaro]

Kostas:
I's also consider -suest-:

 . webuse income

  . regress inc edu exp if male

      Source |       SS           df       MS      Number of obs   =       110
-------------+----------------------------------   F(2, 107)       =     20.05
       Model |  639.919043         2  319.959521   Prob > F        =    0.0000
    Residual |  1707.31485       107  15.9562136   R-squared       =    0.2726
-------------+----------------------------------   Adj R-squared   =    0.2590
       Total |   2347.2339       109  21.5342559   Root MSE        =    3.9945

------------------------------------------------------------------------------
         inc | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
         edu |   1.841002    .383369     4.80   0.000     1.081018    2.600986
         exp |   1.590727   .3569439     4.46   0.000     .8831278    2.298327
       _cons |   1.783822   .3818906     4.67   0.000     1.026769    2.540876
------------------------------------------------------------------------------

 . estimates store Male

 . regress inc edu exp if !male

      Source |       SS           df       MS      Number of obs   =       167
-------------+----------------------------------   F(2, 164)       =     43.30
       Model |  1418.47853         2  709.239266   Prob > F        =    0.0000
    Residual |  2686.09306       164  16.3786162   R-squared       =    0.3456
-------------+----------------------------------   Adj R-squared   =    0.3376
       Total |  4104.57159       166  24.7263349   Root MSE        =    4.0471

------------------------------------------------------------------------------
         inc | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
         edu |   2.475213   .3160483     7.83   0.000     1.851165    3.099261
         exp |   1.354081   .3043211     4.45   0.000     .7531885    1.954974
       _cons |   1.250719   .3132966     3.99   0.000     .6321043    1.869334
------------------------------------------------------------------------------

. estimates store Female

. suest Male Female

Simultaneous results for Male, Female                      Number of obs = 277

------------------------------------------------------------------------------
             |               Robust
             | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
Male_mean    |
         edu |   1.841002   .3911029     4.71   0.000     1.074454    2.607549
         exp |   1.590727   .3320187     4.79   0.000     .9399827    2.241472
       _cons |   1.783822   .3829948     4.66   0.000     1.033166    2.534478
-------------+----------------------------------------------------------------
Male_lnvar   |
       _cons |   2.769848   .1328349    20.85   0.000     2.509497      3.0302
-------------+----------------------------------------------------------------
Female_mean  |
         edu |   2.475213   .3093986     8.00   0.000     1.868803    3.081623
         exp |   1.354081   .2982058     4.54   0.000     .7696084    1.938554
       _cons |   1.250719   .3122779     4.01   0.000      .638666    1.862773
-------------+----------------------------------------------------------------
Female_lnvar |
       _cons |   2.795977   .0976384    28.64   0.000     2.604609    2.987344
------------------------------------------------------------------------------

. test [Male_mean = Female_mean]

 ( 1)  [Male_mean]edu - [Female_mean]edu = 0
 ( 2)  [Male_mean]exp - [Female_mean]exp = 0

           chi2(  2) =    1.79
         Prob > chi2 =    0.4082

.

[Kostas Pappas]

Thank you all for your replies.