Rwolf: repeated numbers in the answer, chance or error

[Antonio Alfonso]

I would like you to report the Romano-Wolf p-values and not the individual t-tests, but when outputting the highly repeated values, it gives me the feeling that it is an error.

Example

clear all
insobs 500
set seed 1
generate dice = round(runiform(1,16),1)
egen float treatment = seq(), from(1) to(2) block(1)
gen subgroup = 0

generate d1 = 0 in 1/100
replace d1 =1 if dice<4 in 1/100
replace subgroup = 1 in 1/100

generate d2 = 0 in 101/200
replace d2 = 1 if dice<4 in 101/200
replace subgroup = 2 in 101/200

generate d3 = 0 in 201/300
replace d3 =1 if dice<4 in 201/300
replace subgroup = 3 in 201/300

generate d4 = 0 in 301/400
replace d4 =1 if dice<7 & treatment== 1 in 301/400 // big difference
replace d4 =1 if dice<4 & treatment== 2 in 301/400

replace subgroup = 4 in 301/400

generate d5 = 0 in 401/500
replace d5 =1 if dice<6 & treatment== 1 in 401/500 //small difference
replace d5 =1 if dice<4 & treatment== 2 in 401/500
replace subgroup = 5 in 401/500

egen float value = rowfirst(d1 d2 d3 d4 d5)

ttest value, by( treatment)
by subgroup, sort : ttest value, by( treatment)

rwolf d1 d2 d3 d4 d5, indepvar(treatment) method(probit) reps(500) verbose seed(1)

[Antonio Alfonso]

Best example.

clear all
insobs 500
set seed 1
generate dice = round(runiform(1,10),1)
egen float treatment = seq(), from(1) to(2) block(1)
gen subgroup = 0

generate d1 = 0  in 1/100
replace d1 =1  if dice<6 in 1/100
replace subgroup = 1  in 1/100

generate d2 = 0  in 101/200
replace d2 = 1  if dice<4 in 101/200
replace subgroup = 2  in 101/200

generate d3 = 0  in 201/300
replace d3 =1  if dice<3 in 201/300
replace subgroup = 3  in 201/300

generate d4 = 0  in 301/400
replace d4 =1  if dice<7 & treatment== 1 in 301/400
replace d4 =1  if dice<3 & treatment== 2 in 301/400

replace subgroup = 4  in 301/400

generate d5 = 0  in 401/500
replace d5 =1  if dice<9 & treatment== 1 in 401/500
replace d5 =1  if dice<2 & treatment== 2 in 401/500
replace subgroup = 5  in 401/500

egen float value = rowfirst(d1 d2 d3 d4 d5)

*ttest value, by( treatment)
*by subgroup, sort : ttest value, by( treatment)

rwolf d1 d2 d3 d4 d5,  indepvar(treatment)  method(probit)   reps(100)  verbose seed(1)

[Margaret Leighton]

Hello Antonio, I'm wondering if you have made in progress in understanding this since your post? I have come across this as well, and am hoping to understand better what is driving it.