Dear Stata-experts,
I am doing a study on psychosocal predictors of a health outcome and was asked by a reviewer to adjust for multiple tests.
From what I have read, bonferroni adjustment is often too conservative, and I am leaning toward the Benjamini-Hochberg procedure using stata commands parmtest and qqvalue.
My question is how exactly to do this with my specific data. I have read the help files, Benjamini, Y. and Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, and several papers by R. Newson, but I haven't been able to find an example that I would know how to transfer to my data - hoping you can help.
Using auto data as an example, I would use the following commands:
sysuse auto, clear
MODEL 1:
parmby "reg length weight trunk headroom rep78 , vce(robust)",by( foreign ) label command saving(test)
use "\test.dta", clear
set more off
qqvalue p, method(bonferroni) qvalue(bonf)
qqvalue p, method(simes) qvalue(simes)
list
This gives me the p-values adjusted by different procedures, but do I do this for each of the the 3 models seperately? Or only for the final model (containing 13 independent variables?)
That is, say I also estimate 2 additional (nested) models, eg.
MODEL 2:
reg length weight trunk headroom rep78 turn displacement , vce(robust)
and
MODEL 3:
reg length weight trunk headroom rep78 turn displacement price mpg , vce(robust)
I would expect the adjusted p-values to differ for models with more paramaters - but in the paper I want to report results for each model - so do I use the parmby command for each model (and get adjusted p-values for each model seperately) - or is this misleading?
Also, in my paper, I first estimate zero-order correlations among all study variables (a total of 14, dichotomous, categorical and continuous) - these are not used for model selection, but should do I still adjust them for multiple hypothesis testing as well?
I hope my question is clear, please ask if not.
Best, Anne
I am doing a study on psychosocal predictors of a health outcome and was asked by a reviewer to adjust for multiple tests.
From what I have read, bonferroni adjustment is often too conservative, and I am leaning toward the Benjamini-Hochberg procedure using stata commands parmtest and qqvalue.
My question is how exactly to do this with my specific data. I have read the help files, Benjamini, Y. and Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, and several papers by R. Newson, but I haven't been able to find an example that I would know how to transfer to my data - hoping you can help.
Using auto data as an example, I would use the following commands:
sysuse auto, clear
MODEL 1:
parmby "reg length weight trunk headroom rep78 , vce(robust)",by( foreign ) label command saving(test)
use "\test.dta", clear
set more off
qqvalue p, method(bonferroni) qvalue(bonf)
qqvalue p, method(simes) qvalue(simes)
list
This gives me the p-values adjusted by different procedures, but do I do this for each of the the 3 models seperately? Or only for the final model (containing 13 independent variables?)
That is, say I also estimate 2 additional (nested) models, eg.
MODEL 2:
reg length weight trunk headroom rep78 turn displacement , vce(robust)
and
MODEL 3:
reg length weight trunk headroom rep78 turn displacement price mpg , vce(robust)
I would expect the adjusted p-values to differ for models with more paramaters - but in the paper I want to report results for each model - so do I use the parmby command for each model (and get adjusted p-values for each model seperately) - or is this misleading?
Also, in my paper, I first estimate zero-order correlations among all study variables (a total of 14, dichotomous, categorical and continuous) - these are not used for model selection, but should do I still adjust them for multiple hypothesis testing as well?
I hope my question is clear, please ask if not.
Best, Anne
