p-hack demonstration

[Andrew Hersh]

I'm giving a talk to med students/residents and wanted to show how easily a researcher can find significance if they play with subgroups, interim analyses, different model specifications...
is anyone aware of an existing program to do something like this?

[Clyde Schechter]

Drat! I had written a program to demonstrate that the last time I taught introductory statistics, but that was a number of years ago and the file has gone lost since then! It's not hard and doesn't even take a lot of thought. You just generate a data set of moderate size and a moderate number of variables with random numbers as data (you can select the means and standard deviations if you want so you can give the variables names that make the data set appear realistic). Then you just script a whole bunch of subset analyses and interim analyses. If you do sufficiently many of them, the law of large numbers will bless you with Type I errors!