Wald test

[Carol Gallo]

Hi everyone,

I'm running a hierarchical model to examine predictors for my outcome (yes/no), in a backward stepwise rationale, following this conceptual approach: https://pubmed.ncbi.nlm.nih.gov/9126524/

I have only categorical independent variables with 3-5 categories each, so after run the regression, I used Wald test for each variable to generate an 'overall' p-value and then check if this p-value is less or equal than 0.20 (to decide about retain or not the variable in the final model).
testparm i.[variable]

I am wondering if that is correct or if actually I have to check the p-value for each level/category of the variable, as this can change the decision about retain or not the variable in the final model.

An example bellow - note that based on Wald test I would not retain this variable, but based on individual p-value I would retain it:


Survey: Poisson regression

Number of strata = 1 Number of obs = 4,507
Population size = 1,274,022
Replications = 100
Design df = 99
F(19, 81) = 8.61
Prob > F = 0.0000

----------------------------------------------------------------------------------------
| Jknife *
OUTCOME | Coefficient std. err. t P>|t| [95% conf. interval]
-----------------------+----------------------------------------------------------------
|
independent_variable_1 |
2 | .1107948 .1343242 0.82 0.411 -.1557337 .3773232
3 | .0821731 .1372108 0.60 0.551 -.1900828 .3544291
4 | .2458391 .120644 2.04 0.044 .0064551 .485223
5 | .1734045 .1234669 1.40 0.163 -.0715806 .4183896
----------------------------------------------------------------------------------------

. testparm i.independent_variable_1

Adjusted Wald test

( 1) [kidwhtR_high]2.q_pca_upp = 0
( 2) [kidwhtR_high]3.q_pca_upp = 0
( 3) [kidwhtR_high]4.q_pca_upp = 0
( 4) [kidwhtR_high]5.q_pca_upp = 0

F( 4, 96) = 1.34
Prob > F = 0.2596

[Felix Bittmann]

To estimate the joint influence, this seems fine to me. However, a few things
1. you can probably use Stata's stepwise command to make this easier and automated.
2. The paper you cite is from 1997. Many fields have by now a different view on how to select relevant variables (e.g. what is your goal? Explain variance? Estimate causal effects?)
3. Especially when your sample is large, this approach based on p-values might be misleading. Approaches based on variance explanation can be more robust.

[Carol Gallo]

Thank you for your reply, Felix!

1) I've just taken a look in stepwise command, and it seems really helpful. It also does the Wald test, but I noticed it remove only specific categories (of a given categorical variable) that aren't significant. So, this leads me to understand that maybe it's better to analyze each category separately.
The only issue is that I have to use jackknife method to survey weight my analyses, and seems it is not allowed in stepwise command.

2) About the paper, it's really old, but this approach has been used in some fields of epidemiological research. I am applying a similar method of this paper: https://www.sciencedirect.com/scienc...65032720331323
My specific goal is to analyze the data in a hierarchical conceptual approach, to find predictors of the outcome (obesity in my case).
If you know some approaches to deal with that, I'm happy to explore...

3) Makes sense the issue about p-value, my sample is big and it's representative. What kind of variance explanation analyses could I explore?

[Felix Bittmann]

1. Interesting how stepwise does this, but this means you need to always recode the categories in the end? Because usually, you want to include the entire variable, not only specific dummies (thats at least what makes sense to me).
2. and 3. This is not really my area of expertise, but the dominance approach can be utilized to decompose variance of correlated variables, regardless of sample size and p-values. In Stata, the user-written command domin is available.

[Andrew Musau]

My specific goal is to analyze the data in a hierarchical conceptual approach, to find predictors of the outcome (obesity in my case).
If you know some approaches to deal with that, I'm happy to explore...

Stepwise procedures have well-known statistical problems. The selected model can be quite unstable across samples, i.e., relatively small changes in the data can produce quite different selected models. In addition, estimated standard errors and p-values do not account for the preceding model-selection process. If your main purpose is model selection and prediction, I would instead look at LASSO or another penalized regression method to stepwise selection. Stata has excellent documentation and commands on this, including worked examples. See

help lasso

[Clyde Schechter]

Having repeatedly denounced the use of -stepwise- on this Forum over many years, I feel a bit strange making this comment. But the concern that -stepwise- "remove only specific categories (of a given categorical variable)" is incorrect. One aspect of -stepwise- is that if a group of variables is enclosed in parentheses, then that group is evaluated as a whole, not variable by variable. Notice the difference in this example:

. sysuse auto
(1978 automobile data)

.  stepwise, pr(.2): regress mpg c.weight##c.weight displ gear turn headroom i.rep78 price
note: 1b.rep78 omitted because of estimability.

Wald test, begin with full model:
p = 0.9874 >= 0.2000, removing displacement
p = 0.8147 >= 0.2000, removing 2.rep78
p = 0.7867 >= 0.2000, removing headroom
p = 0.5486 >= 0.2000, removing 5.rep78
p = 0.3603 >= 0.2000, removing gear_ratio

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(6, 62)        =     26.64
       Model |  1686.24707         6  281.041178   Prob > F        =    0.0000
    Residual |   653.95583        62  10.5476747   R-squared       =    0.7206
-------------+----------------------------------   Adj R-squared   =    0.6935
       Total |   2340.2029        68  34.4147485   Root MSE        =    3.2477

-----------------------------------------------------------------------------------
              mpg | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
------------------+----------------------------------------------------------------
           weight |   -.016902   .0041509    -4.07   0.000    -.0251994   -.0086046
                  |
c.weight#c.weight |   2.32e-06   7.12e-07     3.25   0.002     8.92e-07    3.74e-06
                  |
            price |  -.0004495    .000196    -2.29   0.025    -.0008414   -.0000577
                  |
            rep78 |
               3  |  -1.893132   .9815967    -1.93   0.058    -3.855316    .0690518
               4  |  -2.666823   1.093967    -2.44   0.018    -4.853633   -.4800139
                  |
             turn |  -.4633483   .1931339    -2.40   0.019    -.8494174   -.0772791
            _cons |   72.54151   9.095311     7.98   0.000     54.36024    90.72277
-----------------------------------------------------------------------------------

.  stepwise, pr(.2): regress mpg c.weight##c.weight displ gear turn headroom (i.rep78) price
note: 1b.rep78 omitted because of estimability.

Wald test, begin with full model:
p = 0.9874 >= 0.2000, removing displacement
p = 0.7410 >= 0.2000, removing headroom
p = 0.4045 >= 0.2000, removing gear_ratio

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(8, 60)        =     19.56
       Model |   1691.6591         8  211.457387   Prob > F        =    0.0000
    Residual |  648.543804        60  10.8090634   R-squared       =    0.7229
-------------+----------------------------------   Adj R-squared   =    0.6859
       Total |   2340.2029        68  34.4147485   Root MSE        =    3.2877

-----------------------------------------------------------------------------------
              mpg | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
------------------+----------------------------------------------------------------
           weight |  -.0162724   .0042967    -3.79   0.000     -.024867   -.0076778
                  |
c.weight#c.weight |   2.21e-06   7.38e-07     2.99   0.004     7.33e-07    3.68e-06
                  |
            price |  -.0004608   .0001996    -2.31   0.024    -.0008601   -.0000615
                  |
           rep78 |
               2  |   .2010951   2.623244     0.08   0.939    -5.046174    5.448364
               3  |  -1.159204   2.430152    -0.48   0.635    -6.020231    3.701824
               4  |  -1.815129   2.521809    -0.72   0.474    -6.859499     3.22924
               5  |   1.352864   2.650825     0.51   0.612    -3.949576    6.655304
                  |
             turn |  -.4204784   .2068705    -2.03   0.047     -.834281   -.0066758
            _cons |   69.26458   10.32897     6.71   0.000     48.60356     89.9256
-----------------------------------------------------------------------------------

That said, really, don't use -stepwise-. Follow Andrew Musau's excellent advice.

[Bruce Weaver]

For more info on the problems with the classical algorithmic variable selection methods (including stepwise), see the Stata FAQ on stepwise regression, and see also the Multivariable Modeling Problems section of Frank Harrell's Author Checklist. See especially these sub-sections:

• https://discourse.datamethods.org/t/...e-selection-24
• https://discourse.datamethods.org/t/...final-model-25
• https://discourse.datamethods.org/t/...-validation-26

Finally, I also recommend Mike Babyak's nice (and very readable) 2004 article on overfitting. I think Mike himself might make a few small changes if he was writing that article today--e.g., he would probably increase his recommended "events-per-variable" (EPV) rule of thumb upwards from 10-15 to 20 to accord with Harrell's more recent suggestion. And he might talk about events per model parameter (EPP) rather than EPV to account for categorical variables like yours. But despite a few small issues like that, I think this article remains a useful and accessible overview of the problem of overfitting.

Cheers,
Bruce