Mean statistically testing between models Control vs Treatment

[Eileen Mitchell]

Hello all I am conducting a WTP study I have two groups 1. Control 2. Treatment (Both binary outcomes 0/1)

I want to see if the control WTP mean is statistically different from treatment mean WTP.


Stata commands: Using a Binary Logit model

logit control, bid
wtpcikr bid *gives a mean/ median output*
estimates store A *I think this stores the mean estimates*

logit treatment , bid
wtpcikr bid
estimates store B

My question is, which command in stata to test the means between the control and treatment group is correct:

1. ttest control == treatment, unpaired unequal (testing the binary variables)
2. ttest estimate A == estimate B (testing the stored estimate output of binary variables)

When I run the second test (A==b) - it is significant
Whilst (control==treatment) - it is showing not significant

Which tests looks to be correct? I was thinking the second option but am not 100%.

The observations in my control test are around 300
The observations in my treatment test are around 1000
When I run the second test (A==B) it is showing combined observations of 2600 (i.e it appears to be not focusing on just the control and treatment group).

Thanks for viewing, any help would be great.

[Igor Paploski]

Hello Eileen,

Please consider sharing a snippet of your data using dataex, responses tend to be more frequent when a bit of data is shared. This helps others understand your dataset, see possible issues with it and test solutions/code.

That said, if the objective is to test if the frequency of outcome is equal in both groups (treatment and control), in your case (since both exposure and outcome are dichotomous) both logistic regression and a ttest could work. Consider this hypothetical database:

clear
set obs 100
gen id = _n
gen treat = runiform()
gen outcome = runiform()
replace treat = 1 if treat>=.5
replace treat = 0 if treat<.5
replace outcome = 1 if outcome>=.5
replace outcome = 0 if outcome<.5
label define treat 0 "Control" 1 "Treatment"
label values treat treat

*If you run a logit/logistic model, you get the odds of having the outcome in the treatment group when compared to being in the control group, along with its p-value and 95% CI.
logit outcome treat, or

*you could try doing this test using a ttest, such as:
ttest outcome, by(treat)

[Tiago Pereira]

If you have two independent groups and a binary variable, why not a test comparing proportions?

[Eileen Mitchell]

HI Tiago and Igor, many thanks for your responses. I have attached a few snips of the code.


Logit control1, bid

n=340
log likelihood=-229.864

Variable	coef	std.err
bid	-0.161	0.647
cons	0.091	0.17

Next I type into stata the following command to get the mean

wtpcikr bid

Measure	WTP	LB	UP	ASL
Mean	0.57	-4.87	2.01	0.2958

I then type estimates store A


Next I run the regression for the treatment group


Logit treat, bid

n= 715
log likelihood=-483.748

Variable	coef	std.err
bid	-0.144	044
cons	0.4377	0.12

wtpcikr bid

Measure	WTP	LB	UP	ASL
Mean	0.30	-2.96	1.42	0.3648

I then type estimates store B



Next I run the ttests - and hence where I am unsure which one is correct.


I want to test the that the mean output i.e

WTP=0.57 in the control is statistically different from the mean WTP= 0.30 in the treatment group.


1.
ttest control1 == treat, unpaired unequal (original binary options)

Variable	obs	mean	std err
Control Treat	340 715	0.438 0.433	0.0269 0.0185
Combined	1055	0.435	0.152
diff		0.004	0.0327

diff= mean(estimate A) - mean (estimate B)
Ho: diff=0
t=0.1427
p=0.8866


2.
ttest _est_A== _est_B, unpaired unequal (stored estimates)

Variable	obs	mean	std err
Control Treat	1424 1424	0.238 0.502	0.11 0.013
Combined	2848

diff= mean(estimate A) - mean (estimate B)
Ho: diff=0
t=-15.1184
p= 0.0000

[Igor Paploski]

Hello Eileen,

I don't know what the bid option on your logit (not Logit) model is. I couldn't find any bid option to the logit command under the logit help. I assume it might be related to the Krinsky and Robb CI (which I also never used)? This seems like a problem I can't give advice on. There are other topics on Statalist asking for info on WTP issues (see here, here and here).

[Eileen Mitchell]

hi Igor, the bid option just relates to the amount of bids presented to respondents ranging from 0.50 - 5.00 euros. and if they would be willing to pay these amounts. Yes, the command wtpcikr relates to the krinsky and Robb CI.

I was just trying to see which method was best to compare my mean estimates - I found something online that suggest grouping the treatments and then comparing them. So I will try that to see does it work.

Thanks for your help

[Saika Belal]

Hi Eileen, I am facing the same issue as you. I'm wondering if you found a solution? How did you group the treatments to compare them?

Thanks!

[Rocio Nunez]

Hi, I'm facing a similar problem. The thing is I have 4 groups (One control and three treatments). So since I randomly chose the subjects for each group I need to know if the difference in mean are statistically significant. Is there a way to compare control vs the first treatment by age or sex, and so on with the other treatments?

[Felix Bittmann]

Stata provides a nice introduction for this topic: https://www.stata.com/features/overv...e-comparisons/