FWER Corrected p-values (Multiple Hypothesis Test)

[Steven Ji]

Dear Statalisters,

I hope you are doing very well. I am doing a multiple hypothesis test correction to p-values using
FWER corrected p values (List, Shiekh, 2019).
I have 8 outcomes and 4 treatment groups (+1 control group).
I want to specify my outcomes and treatment groups in a single-family and see how corrected p-values increase? My intuition is as the number of hypotheses tested increases, my p-value should also increase, however, I am not getting that result which most likely means I am doing something wrong. Also, my p values are incredibly small, making me question my method.

Therefore, I want to ask
- how to specify here via MHTEXP or via an alternate command number of hypothesis test in the multiple hypothesis tests to make the FWER correction.

Your help or any leads on this will be really helpful since I am unable to figure this out for quite a few days now.

My sample data and the exact codes I tried I use are as follows

gen treatment =.
replace treatment=1 if treatment_1==1
replace treatment=2 if treatment_2==1
replace treatment=3 if treatment_3==1
replace treatment=4 if treatment_4==1


mhtexp outcome_1 outcome_2 outcome_3 outcome_4 $controls, treatment(treatment) bootstrap(1000) // WITH and WITHOUT CONTROLS

mhtexp outcome_1 outcome_2 outcome_3 outcome_4 outcome_5 outcome_6 outcome_7 outcome_8 $controls, treatment(treatment) bootstrap(1000) // WITH and WITHOUT CONTROLS

* Example generated by -dataex-. To install: ssc install dataex
clear
input byte(treatment_1 treatment_2 treatment_3 treatment_4 placebo outcome_6 outcome_5) double(outcome_7 outcome_4 outcome_3 outcome_2 outcome_1 outcome_8) byte(control_1 control_2 control_3 control_4 control_5 control_6 control_7 control_8) int cluster
0 1 0 0 0 3 0    .1653932764197893 -.17467090552115214 -.3578288676205426 .032402727645159196       -.26534503698349 -.2238087420821282  2 30 33 1 10 0 0 0   1
0 0 1 0 0 5 1  -.19725712256929778 -.17467090552115214 -.3578288676205426   .4015287542951884    -.13001234829425812 -.2238087420821282  2 30 40 1 10 0 1 1   2
0 1 0 0 0 3 0   .13517240983736548 -.17467090552115214 -.3578288676205426  -.9879681864926527     -.2359367161989212 -.2238087420821282  5 30 50 0 12 0 1 0   3
0 0 0 1 0 1 0 -.015931923074754238 -.17467090552115214 -.3578288676205426 -.40696546501426895    -.22635124623775482 -.2238087420821282  5 30  1 0 14 0 1 1   4
0 0 0 1 0 1 0   .10495154325494163 -.17467090552115214 2.7900279823166945  -.4222489794071104      .3407953381538391 -.2238087420821282  6 20  6 0 14 1 1 1   5
0 0 0 0 1 2 0  -2.7660307820753327 -.17467090552115214 -.3578288676205426                   .     -.1357412189245224 -.2238087420821282  3 30 40 1 12 0 1 1   6
1 0 0 0 0 2 0   -.2879197223165698 -.17467090552115214 -.3578288676205426   .6973684850587097    -.21412818133831024 -.2238087420821282  5 30  4 1 12 0 0 0   7
0 0 0 0 1 2 0   -.7714535876353528 -.17467090552115214 2.7900279823166945   .9442699696663021    -.07100307196378708 -.2238087420821282  4 30 30 1 12 0 1 1   8
0 1 0 0 0 1 0    .7093688749034205 -.17467090552115214 -.3578288676205426   -.937453325868029     -.2908829152584076 -.2238087420821282 12 30 30 0 12 0 1 1   9
0 1 0 0 0 3 0    .3467184759143334 -.17467090552115214 -.3578288676205426   -.797864071810256    -.14190490543842316 -.2238087420821282  2 30 30 0 14 0 1 1  10
1 0 0 0 0 4 0  -.10659452282202574 -.17467090552115214 -.3578288676205426 -1.2565983455268217     -.3622598350048065 -.2238087420821282  3 30  2 0 12 0 1 1  11
0 0 0 0 1 3 0   .28627674274948517 -.17467090552115214 -.3578288676205426                   .   .0036175954155623913 -.2238087420821282  7 30 25 1 14 1 1 1  12
1 0 0 0 0 3 0   -.9527787871298963 -.17467090552115214 -.3578288676205426  .05890282286359865    -.32850730419158936 -.2238087420821282  4 30 46 1 14 0 1 1  13
0 0 0 1 0 1 0    -.650570121305657  5.7156201862199225 -.3578288676205426   .6747011595708129     .31842613220214844 -.2238087420821282  8 30 40 0 12 0 1 1  14
0 1 0 0 0 1 0   2.4621791366840093 -.17467090552115214 -.3578288676205426  -.7389419311622704    .014530192129313946  4.460739755981727  5 30 40 0 14 1 1 1  15
0 0 1 0 0 5 1   -.4692449218111134 -.17467090552115214 -.3578288676205426  .49515703563781094     .45657020807266235 -.2238087420821282  3 30  1 1 12 0 1 0  16
0 0 0 1 0 3 0    .3467184759143334  5.7156201862199225 2.7900279823166945  2.3054773245518265      1.084456205368042 -.2238087420821282  3 30 33 0 14 0 0 0  18
0 1 0 0 0 3 0  -.16703625598687394 -.17467090552115214 2.7900279823166945  .21587541560553616    -.35587406158447266 -.2238087420821282  4 30 27 0 12 0 0 0  17
0 0 0 0 1 3 0    -.650570121305657 -.17467090552115214 -.3578288676205426   .5764800738856288     -.3581002652645111 -.2238087420821282  7 30 23 0 14 0 0 0  19
0 0 0 0 1 5 0   -.9527787871298963 -.17467090552115214 -.3578288676205426   1.213643864977265    -.22128960490226746 -.2238087420821282  3 30 40 0 14 0 1 1  20
0 0 1 0 0 5 1    .6489271417385728 -.17467090552115214 -.3578288676205426                   .      .4726555347442627 -.2238087420821282  3 30  1 0 12 1 0 0  21
1 0 0 0 0 3 0   .46760194224402873 -.17467090552115214 -.3578288676205426 -.44450818127621194    -.19550882279872894  4.460739755981727  2 30  4 1 12 0 0 0  22
1 0 0 0 0 2 0     .316497609331909 -.17467090552115214 -.3578288676205426  -.3967021704897849     .13535897433757782 -.2238087420821282  4 30 17 1 14 0 1 1  23
0 0 0 1 0 3 0   -.5599075215583849  5.7156201862199225 -.3578288676205426 -.06835524381064445    -.06982377916574478 -.2238087420821282  8 30 50 0 14 0 0 0  24
1 0 0 0 0 2 0    1.646215738958563 -.17467090552115214 2.7900279823166945 -.31098124268393185      .1388903707265854 -.2238087420821282  1 30 65 0 14 0 1 1  25
0 1 0 0 0 1 0    .6791480083209966 -.17467090552115214 -.3578288676205426  .18168217354185456      .1321992129087448 -.2238087420821282  4 30  1 0 10 1 0 0  26
0 0 0 1 0 1 0   1.7670992052882584  5.7156201862199225 -.3578288676205426 -.26092606163646775     1.4831353425979614 -.2238087420821282  5 30 52 1 16 1 0 0  27
0 0 0 1 0 3 0   1.8879826716179537  5.7156201862199225 -.3578288676205426 -.37322358705059144      .5248804092407227 -.2238087420821282  4 30 50 1 14 1 1 1  28
0 0 1 0 0 5 1    .8604732078155402 -.17467090552115214 -.3578288676205426   .2923209275729863    -.25364482402801514 -.2238087420821282 10 30 50 1 12 0 0 0  29
0 0 1 0 0 5 1  -1.1945457197892877 -.17467090552115214 -.3578288676205426   .9310766336112833    -.14429421722888947 -.2238087420821282  1 30  1 0 10 0 0 0  30
1 0 0 0 0 1 0   -1.164324853206864 -.17467090552115214 -.3578288676205426  -.1668134869043051      .9537391066551208 -.2238087420821282  4 30 50 1 14 0 1 1  31
0 1 0 0 0 1 0     .316497609331909 -.17467090552115214 -.3578288676205426 -1.0388126726317228    -.33852890133857727 -.2238087420821282  2 30  4 0 12 0 0 0  32
0 1 0 0 0 2 0  -2.3731595165038217 -.17467090552115214 -.3578288676205426 -1.0158000885323273     .30001211166381836 -.2238087420821282  6 30 85 1 14 1 0 0  33
0 0 0 1 0 3 0   1.6159948723761388  5.7156201862199225 -.3578288676205426  -.6680418446003231    -.22986406087875366 -.2238087420821282 12 30 25 0 10 0 1 1  34
1 0 0 0 0 1 0 -.015931923074754238 -.17467090552115214 2.7900279823166945  -.3605893836795129   -.030210819095373154 -.2238087420821282  6 30 30 1 14 0 1 1  35
1 0 0 0 0 4 0   .19561414300221366 -.17467090552115214 -.3578288676205426 -.29625386740325454   -.054053060710430145 -.2238087420821282  2 30  1 0 10 0 1 1  36
1 0 0 0 0 2 0  -.04615278965717807 -.17467090552115214 -.3578288676205426   .0974127954282769      -.416740357875824 -.2238087420821282  4 30  1 1 12 0 1 1  37
1 0 0 0 0 3 0   .46760194224402873 -.17467090552115214 -.3578288676205426  -.4155743122038819      -.335686594247818 -.2238087420821282  5 30 35 0 10 0 0 0  38
0 0 0 1 0 3 0   -.7412327210529289  5.7156201862199225 -.3578288676205426                   . -.00035469839349389076  4.460739755981727  3 35  3 1 14 0 1 1  39
0 0 0 1 0 1 0    .7395897414858443 -.17467090552115214 -.3578288676205426   .2073853134297047   -.027304796501994133 -.2238087420821282  4 30  1 0 12 0 1 1  40
0 1 0 0 0 1 0   -.7714535876353528 -.17467090552115214 -.3578288676205426  .21471124880375708    -.23395533859729767 -.2238087420821282  3 30  3 0 12 0 1 1  41
0 0 0 1 0 1 0   1.6159948723761388 -.17467090552115214 2.7900279823166945  .21967625390380163      .5807830095291138 -.2238087420821282  5 30  5 0 14 0 1 1  42
0 0 0 0 1 3 0  -1.2549874529541358 -.17467090552115214 -.3578288676205426 -2.0620597440081014     .25282812118530273 -.2238087420821282  5 30  5 0 14 0 1 0  43
1 0 0 0 0 3 0   -.5599075215583849 -.17467090552115214 -.3578288676205426 -.26128395977324326    -.43430620431900024 -.2238087420821282  3 30  2 0 12 0 0 0  44
1 0 0 0 0 4 0    .6187062751561484 -.17467090552115214 -.3578288676205426 -.16187575656933087    -.24444033205509186 -.2238087420821282 10 30 30 1 12 0 1 1  45
0 0 0 0 1 1 0  -.25769885573414547 -.17467090552115214 -.3578288676205426 -1.8413749631574865      .0203559510409832 -.2238087420821282  1 30 65 0 12 0 1 1  46
0 0 1 0 0 5 1  -.10659452282202574 -.17467090552115214 -.3578288676205426 -1.7443014133197707     .17674283683300018 -.2238087420821282  5 30 35 0 14 1 0 1  47
1 0 0 0 0 3 0    -.650570121305657 -.17467090552115214 -.3578288676205426                   .      -.292697012424469 -.2238087420821282  3 30 35 0 12 0 0 0  48
1 0 0 0 0 3 0  -1.0132205202947442 -.17467090552115214 -.3578288676205426  .36018952378310093      .1786084771156311 -.2238087420821282  5 30  4 1 14 0 1 1  49
0 0 1 0 0 5 1   .46760194224402873 -.17467090552115214 -.3578288676205426   .6299302979461671    .009654278866946697  4.460739755981727  7 30 36 0 12 0 1 1  50
0 1 0 0 0 5 0    .5582645419913007 -.17467090552115214 -.3578288676205426  -.9192996413801727     -.3709001839160919 -.2238087420821282  8 30 50 0 14 1 1 1  51
0 1 0 0 0 1 0    .6791480083209966 -.17467090552115214 -.3578288676205426 -1.0516467800736213      .8933067321777344 -.2238087420821282  3 30 38 1 14 1 1 1  52
0 0 0 0 1 2 0  -.19725712256929778 -.17467090552115214 -.3578288676205426  1.6131483253910892    -.23106208443641663 -.2238087420821282  5 30  2 1 12 0 0 0  53
0 0 0 0 1 1 0    .0747306766725178 -.17467090552115214 -.3578288676205426 -.19094034494254644     -.2653968334197998 -.2238087420821282  1 30 35 0 12 0 1 0  54
0 0 1 0 0 4 1    .7395897414858443 -.17467090552115214 -.3578288676205426                   .    .058907583355903625 -.2238087420821282  9 30 40 0 14 0 0 0  55
0 0 1 0 0 3 1     .528043675408877 -.17467090552115214 -.3578288676205426  -1.251316354437607    -.08203176409006119 -.2238087420821282  3 30  5 1 12 1 0 0  56
0 0 1 0 0 5 1   .04450981009009397 -.17467090552115214 -.3578288676205426  -2.620993189115617     -.4887867569923401 -.2238087420821282  5 30 35 1 10 0 0 1  57
0 0 0 1 0 1 0     2.00886613794765 -.17467090552115214 -.3578288676205426   .2614884687564977     .33124205470085144 -.2238087420821282 13 30 50 1 14 0 1 0  58
0 0 1 0 0 3 1    .6187062751561484 -.17467090552115214 -.3578288676205426  1.3454771071824996    -.07964923977851868 -.2238087420821282  2 30 40 0 12 0 1 1  59
0 0 1 0 0 5 1   -.8318953208002005 -.17467090552115214 -.3578288676205426   .5116512835045026     -.2557803988456726  4.460739755981727  4 30  4 1 14 1 1 1  60
0 1 0 0 0 1 0    .3467184759143334 -.17467090552115214 -.3578288676205426 .010749225132069011      -.513462245464325 -.2238087420821282  4 30  3 1 14 0 0 0  61
0 1 0 0 0 3 0  -.43902405522868954 -.17467090552115214 -.3578288676205426  -.8343959586691686      .1515323519706726 -.2238087420821282  2 30 30 0 10 0 1 0  62
0 0 0 0 1 3 0    .8604732078155402 -.17467090552115214 2.7900279823166945    .261200420627829    -.24141129851341248 -.2238087420821282  3 30 30 1 14 0 1 0  63
0 1 0 0 0 1 0    .5884854085737246 -.17467090552115214 -.3578288676205426  -.7536147569128072   .0011115926317870617 -.2238087420821282  5 30  5 1 14 0 1 0  64
0 0 1 0 0 3 1   -.4692449218111134 -.17467090552115214 -.3578288676205426   .3072578528779839     .18833111226558685 -.2238087420821282  6 30  1 0 14 1 0 0  65
0 0 0 0 1 2 0  -.16703625598687394 -.17467090552115214 -.3578288676205426                   .    -.21167317032814026 -.2238087420821282  4 30 40 1 14 0 0 0  66
0 1 0 0 0 2 0   -.8318953208002005 -.17467090552115214 -.3578288676205426  -.8765850309465549     -.4903718829154968 -.2238087420821282  4 30  5 1 12 0 0 1  67
0 1 0 0 0 1 0    -.650570121305657 -.17467090552115214 -.3578288676205426 -1.0488773935617322    -.47122159600257874 -.2238087420821282  2 30 50 1 14 0 1 1  68
0 0 1 0 0 5 1   .46760194224402873 -.17467090552115214 -.3578288676205426    .880124711507363      .5999096632003784 -.2238087420821282  4 30  1 1 12 0 1 1  69
0 0 0 0 1 1 0     .890694074397964 -.17467090552115214 -.3578288676205426                   .     -.3696211874485016 -.2238087420821282  6 30 40 1 10 0 1 0  70
0 0 0 1 0 1 0  -.43902405522868954 -.17467090552115214 2.7900279823166945   -.416889488070806    -.23398828506469727 -.2238087420821282  1 30 31 1 14 1 1 1  71
0 0 0 1 0 1 0  -1.7385213182729187 -.17467090552115214 -.3578288676205426                   .     -.5392345786094666 -.2238087420821282 16 30 95 0 14 1 0 0  72
1 0 0 0 0 1 0  -1.2549874529541358 -.17467090552115214 -.3578288676205426 -.20078254370387266   -.003632044419646263 -.2238087420821282  6 30 38 1 10 0 1 1  73
1 0 0 0 0 1 0   .10495154325494163 -.17467090552115214 2.7900279823166945 -.16739091363798753     .00923280231654644 -.2238087420821282  7 30 42 0 10 0 1 1  74
0 0 0 1 0 1 0  -.37858232206384135 -.17467090552115214 2.7900279823166945  1.3151222893738266     .42960280179977417 -.2238087420821282  5 30 76 0 12 0 0 1  75
1 0 0 0 0 2 0   .10495154325494163 -.17467090552115214 -.3578288676205426 -.49060619305488595     .11721663922071457 -.2238087420821282  4 30  4 0 12 0 1 1  76
0 0 0 0 1 1 0  -1.1945457197892877 -.17467090552115214 -.3578288676205426                   .    -.44221267104148865 -.2238087420821282  4 30  3 1 12 0 0 0  77
0 1 0 0 0 3 0    .1653932764197893 -.17467090552115214 -.3578288676205426 -1.0794287892503627    -.27576348185539246 -.2238087420821282  5 30 47 0 12 0 0 0  78
0 0 0 0 1 2 0    .6187062751561484 -.17467090552115214 -.3578288676205426   .1993861570251378    -.44715332984924316 -.2238087420821282  4 30 45 1 14 0 1 1  79
1 0 0 0 0 3 0  .014288943507669597 -.17467090552115214 -.3578288676205426   .6140776722967988    -.12049747258424759 -.2238087420821282  9 30 75 1 14 1 0 0  80
0 0 0 1 0 1 0   .19561414300221366 -.17467090552115214 -.3578288676205426   .2346018701598676     .08459015935659409 -.2238087420821282  4 30 32 0 10 1 1 1  81
0 0 1 0 0 2 1   1.0417984073100837 -.17467090552115214 -.3578288676205426  -.9620688013439311     .43692710995674133 -.2238087420821282  8 30 40 0 14 0 1 0  82
0 0 0 1 0 1 0  -1.2247665863717119 -.17467090552115214 2.7900279823166945  -.4881158739563375     .14259998500347137 -.2238087420821282  3 30  2 1 10 0 0 0  83
0 0 0 0 1 1 0   -.8923370539650487 -.17467090552115214 -.3578288676205426 -.44843940625627665    -.34136369824409485 -.2238087420821282  7 30 40 0 14 0 1 1  84
0 0 0 1 0 3 0   -.2879197223165698 -.17467090552115214 -.3578288676205426 -1.6741124663641676      .3123893439769745 -.2238087420821282  5 30 30 0 12 0 0 0  85
0 0 0 1 0 3 0   .28627674274948517  5.7156201862199225 -.3578288676205426 -1.3612921986298971    -.24287857115268707 -.2238087420821282  7 30 35 1 14 0 0 0  86
1 0 0 0 0 1 0   .04450981009009397 -.17467090552115214 -.3578288676205426                   .     -.3046002686023712 -.2238087420821282  5 30 53 0 14 0 0 0  87
0 0 0 0 1 2 0   -.7110118544705045 -.17467090552115214 -.3578288676205426  .22056734043579188    -.11875037848949432 -.2238087420821282  6 30 50 1 14 1 1 1  88
0 0 0 0 1 1 0    .8302523412331163 -.17467090552115214 -.3578288676205426   -.568260442781669     .40828466415405273 -.2238087420821282 14 30 23 1 14 1 1 1  89
1 0 0 0 0 2 0  -.16703625598687394 -.17467090552115214 -.3578288676205426   .7923315667943434     .03317645192146301 -.2238087420821282  2 30  5 0 10 0 1 0  90
1 0 0 0 0 2 0  -.22747798915172163 -.17467090552115214 -.3578288676205426                   .     -.2622317373752594 -.2238087420821282  5 30 50 1 14 1 0 0  91
1 0 0 0 0 1 0  -.04615278965717807 -.17467090552115214 -.3578288676205426 .027134041319393423     .17080353200435638 -.2238087420821282  3 30  1 0 14 0 0 0  92
0 0 0 0 1 1 0    .6791480083209966 -.17467090552115214 -.3578288676205426   .8440870966105775    -.08979001641273499 -.2238087420821282  5 30 45 1 14 0 1 1  93
0 0 0 0 1 2 0   -.6203492547232331 -.17467090552115214 2.7900279823166945  .03007073523165263      .5065498352050781 -.2238087420821282  4 30  4 1 14 0 0 0  94
0 0 0 0 1 2 0   -.6203492547232331 -.17467090552115214 -.3578288676205426  1.0573275296924445    -.25364482402801514 -.2238087420821282  1 30  2 0 10 0 1 1  95
0 0 0 1 0 1 0    .7093688749034205 -.17467090552115214 2.7900279823166945                   .      .3561297059059143 -.2238087420821282  8 30 40 1 14 0 1 1  96
0 0 0 0 1 1 0   -.2879197223165698 -.17467090552115214 -.3578288676205426  -.6081657524130509    -.11607611179351807 -.2238087420821282  2 30 40 0 10 1 1 1  97
0 0 0 1 0 1 0   1.2533444733870516 -.17467090552115214 -.3578288676205426  1.7729807770363684     -.3061423897743225  4.460739755981727  5 30 32 0 12 0 1 1  98
0 0 0 1 0 1 0    .6791480083209966 -.17467090552115214 2.7900279823166945                   .    -.14963358640670776 -.2238087420821282  6 30  5 0 12 1 1 1  99
1 0 0 0 0 3 0  -.16703625598687394 -.17467090552115214 -.3578288676205426 -.34600503460591125      .3695245385169983 -.2238087420821282  6 30  5 0 12 0 0 0 100
end

[Roger More]

I have the same question, issue here with p values correcting for multiple hypothesis is smaller than the standard p-values. The reference on FWER and q values here does not help either: Anderson, M.L., 2008. Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects. Journal of the American statistical Association, 103(484), pp.1481-1495. Statalisters help here would really be appreciated. Thank you.

[Mead Over]

Steven Ji : Your intuition seems correct to me. Here are some suggestions that you may have already tried.

Have you tried working through the examples in the -mhtexp- help file to see if they confirm your intuition? Have you tried consulting Seidel's GitHub page on this program here: https://github.com/seidelj/mht ? If I were trying to answer your question, I would try to use your data to perform multiple hypothesis testing on a simplified version of your model. Perhaps try to use your data to estimate a model which is as close as possible to Seidel's own examples. If you can get MHT to work as you expect on a simpler version of your model, you can then add model elements one at a time until the results no longer match your intuition. Perhaps this process will reveal what's going on. Hope these suggestions help.

[Mead Over]

In both of your questions, you imply that a p-value should get worse (i.e. larger) when the number of hypotheses tested increases. While this prediction matches our intuition and will generally hold when the covariates are distributed multivariate normal and all are mutually independent, the intuition may be misleading when the covariances between pairs of variables are not zero. See
https://www.stat.cmu.edu/~larry/=stat401/lecture-18.pdf