p-value difference between regress and ttest

[Elizabeth Park]

Hello, I am grappling with a question and would like some advice from the statalist community. My colleague and I are working on cleaning and analyzing survey data right now, and to get some descriptives, we are running a lot of ttests. We are running multiple-group ttests (we have multiple treatment groups rather than just two groups), therefore we are using the command ttesti (which, you may know, you input the sample size, standard deviation and the mean manually to test multiple groups). We are running the difference in group means for four groups which makes it six different group combination (1vs2, 1vs 3, 1vs4, 2vs3, 2vs4 3vs4). Instead of using the command ttesti, we have also tried using the regress command (i.e., "regress y i.group") and using the postestimation command "pwcompare..., pveffects" to get multiple group comparisons p-values.

So, our question that we are grappling with is, the p-values that we get when we use ttesti/ttest and regress are different and we do not know why. We suspect it may be due to using "i." perhaps but I couldn't find more info on this on the web.

Below are the specific codes as an example:

sum y if group==1&CAMPUS==0
sum y if group==3&CAMPUS==0

ttesti 85 3304.612 3163.371 62 2716.484 3175.046, welch // p-value = 0.2686
regress y ib1.group if CAMPUS==0 // p-value = 0.261

Does anyone have any idea what could be going on?

Thank you!
Liz

[Carole J. Wilson]

You are using the -welch- option, so therefore an assumption of unequal variances. The regression assumption is that the variances are equal, unless you use a robust estimator. Comparing the results of your unequal variance t-test with reg y ib1.group, robust should get your p-values closer, and they will converge with large N's.

[Marcos Almeida]

Carole gave an insightful explanation. That said, I think we could profit from this opportunity shared by Elizabeth so as to underline (again) the issue on "too much" value towards p-values. In this case, both (rounded) p-values are 0.26.

[Clyde Schechter]

And from the pedantry corner:

and to get some descriptives, we are running a lot of ttests

is pretty much an oxymoron. If you are using the t-test as a cumbersome and inconvenient way of getting the N's means and standard deviations (which you could more easily get with -tabstat- or even -summarize-) then fine. But t-statistics and p-values are inferential statistics, not descriptive statistics. That is, they are used to make inferences about population parameters from sample data. Descriptive statistics are used only to describe the sample at hand.

[Elizabeth Park]

Thanks so much for the advice Carol and Marcos. And in response to Clyde, yes that was a mistake in writing on my part -- thanks for catching it.