Data Analysis

[Tony Josipovic]

Hello,

I have determined 2 parameters (mean_alpha and mean_delta) using 6 different methods for each participant. Now I need to find out if I can get the same values for the 2 parameters using the 6 methods. I used ttest and thought that p value must be insignificant for the values to be the same between methods. I used the code (ttest mean_alpha if method_nr....) to test mean_alpha (analogously also for mean_delta). Unfortunately, the p-values do not show that there is no difference between the methods. Is my analysis wrong? How can I still compare the methods? Can anyone please tell me what else I could do. Also, I did the same survey with the same participants after one month to see if the results are time-consistent. What is the best way to analyze this? Unfortunately, I have never done this before.

ttest mean_alpha if method_nr==1 |method_nr==2, by (method_nr) // p=0.0192
ttest mean_alpha if method_nr==1 |method_nr==3, by (method_nr) // p=0.0111
ttest mean_alpha if method_nr==1 |method_nr==4, by (method_nr) // p=0.1467
ttest mean_alpha if method_nr==1 |method_nr==5, by (method_nr) // p=0.0082
ttest mean_alpha if method_nr==1 |method_nr==6, by (method_nr) // p=0.1253

ttest mean_alpha if method_nr==2 |method_nr==3, by (method_nr) // p=0.0000
ttest mean_alpha if method_nr==2 |method_nr==4, by (method_nr) // p=0.4004
ttest mean_alpha if method_nr==2 |method_nr==5, by (method_nr) // p=0.7710
ttest mean_alpha if method_nr==2 |method_nr==6, by (method_nr) // p=0.1512

ttest mean_alpha if method_nr==3 |method_nr==4, by (method_nr) // p=0.0001
ttest mean_alpha if method_nr==3 |method_nr==5, by (method_nr) // p=0.0000
ttest mean_alpha if method_nr==3 |method_nr==6, by (method_nr) // p=0.1034

ttest mean_alpha if method_nr==4 |method_nr==5, by (method_nr) // p=0.2585
ttest mean_alpha if method_nr==4 |method_nr==6, by (method_nr) // p=0.1415


Furthermore, I tried also the following code, since it is shorter and nicer:

pwmeans mean_alpha, over(method_nr) pveffects


but then I got different p-values. For example, when I compare method 1 with the other methods, with this code it is significantly different only for method 6 and with the first code (ttest mean_alpha if method_nr==....) it was significantly different for method 2, 3, 5.




Thanks.

[George Ford]

search dataex

[Tony Josipovic]

search dataex

You mean, I should post a data example?

[Fei Wang]

Tony, pwmean assumes equal variance across all six groups, while ttest assumes equal variance only for the two groups you are comparing. That's why you have different results. If you only have two groups in total, both commands will give identical results.

However, both methods, in my view, are inappropriate for your case, because they assume independent samples across groups while your case apparently is a matched sample -- six measures are constructed for the same group of people (they can't be independent). A better way is to run a paired t-test for each pair of measures.

[Felix Bittmann]

Interesting question. You can easily rebuild this problem with generic data.

clear all
version 16.1
sysuse nlsw88
drop if missing(race, hours)


* All pairwise contrasts*
recode race (1 = 0) (3 = 1) (else = .), gen(v1)
recode race (1 = 0) (2 = 1) (else = .), gen(v2)
recode race (2 = 0) (3 = 1) (else = .), gen(v3)


ttest hours, by(v1)
ttest hours, by(v2)
ttest hours, by(v3)
pwmean hours, over(race) pveffects

[Tony Josipovic]

That is dataex

[Fei Wang]

(#4 continued)

Your data seem to be in long form, and paired t-tests require wide form. You may need to reshape your data first.

reshape wide mean_alpha mean_delta, i(ParticipantID) j(method_nr)

Then, you may run the paired t-test, for example, to compare mean_alpha from measure 1 to that from measure 2.

ttest mean_alpha1 == mean_alpha2

There may be a way of conducting all the paired t-tests together (like pwmean), but it's beyond my knowledge.

[Tony Josipovic]

Tony, pwmean assumes equal variance across all six groups, while ttest assumes equal variance only for the two groups you are comparing. That's why you have different results. If you only have two groups in total, both commands will give identical results.

However, both methods, in my view, are inappropriate for your case, because they assume independent samples across groups while your case apparently is a matched sample -- six measures are constructed for the same group of people (they can't be independent). A better way is to run a paired t-test for each pair of measures.

Thanks a lot for explaining it. Now I see why the results are different. When I want to conduct a paired ttest, I write this code:

ttest mean_alpha if method_nr==1 == ttest mean_alpha if method_nr==2
ttest not found
r(111);

but I see that ttest is tretaed as a variable. How can I conduct the paired ttets correctly?

[Felix Bittmann]

Did you reshape as explained in post #7? Try

encode method, gen(m)
ssc install fre, replace
fre m
keep m mean_alpha mean_delta user_id

reshape wide mean_alpha mean_delta, i(user_id) j(m)

ttest mean_alpha1 == mean_alpha6    //1 VS 6
ttest mean_alpha1 == mean_alpha5   //1 VS 5
...

[Tony Josipovic]

Did you reshape as explained in post #7? Try

encode method, gen(m)
ssc install fre, replace
fre m
keep m mean_alpha mean_delta user_id

reshape wide mean_alpha mean_delta, i(user_id) j(m)

ttest mean_alpha1 == mean_alpha6 //1 VS 6
ttest mean_alpha1 == mean_alpha5 //1 VS 5
...

Hi Felix,

with the code reshape as explained in post #7 I got

. reshape wide mean_alpha mean_delta, i(user_id) j(method_nr)
(note: j = 1 2 3 4 5 6)
variable method not constant within user_id
variable switch_1 not constant within user_id
variable switch_2 not constant within user_id
variable SVERWEIS not constant within user_id
variable minalpha not constant within user_id
variable maxalpha not constant within user_id
variable mindelta not constant within user_id
variable maxdelta not constant within user_id
Your data are currently long. You are performing a reshape wide. You typed something like

. reshape wide a b, i(user_id) j(method_nr)

There are variables other than a, b, user_id, method_nr in your data. They must be constant within user_id because that is
the only way they can fit into wide data without loss of information.

The variable or variables listed above are not constant within user_id. Perhaps the values are in error. Type reshape error
for a list of the problem observations.

Either that, or the values vary because they should vary, in which case you must either add the variables to the list of xij
variables to be reshaped, or drop them.
r(9);

Now I tried your code and I get this:

encode method, gen(m)

. ssc install fre, replace
connection timed out -- see help r(2) for troubleshooting
http://fmwww.bc.edu/repec/bocode/f/ either
1) is not a valid URL, or
2) could not be contacted, or
3) is not a Stata download site (has no stata.toc file).
r(2);

[Felix Bittmann]

Yeah thats why I deleted these vars first to avoid this error message. Regarding fre this is just for convenience so try:

encode method, gen(m)
tab m
keep m mean_alpha mean_delta user_id

reshape wide mean_alpha mean_delta, i(user_id) j(m)

ttest mean_alpha1 == mean_alpha6    //1 VS 6
ttest mean_alpha1 == mean_alpha5   //1 VS 5
...

[Fei Wang]

Hi Felix,

with the code reshape as explained in post #7 I got

. reshape wide mean_alpha mean_delta, i(user_id) j(method_nr)
(note: j = 1 2 3 4 5 6)
variable method not constant within user_id
variable switch_1 not constant within user_id
variable switch_2 not constant within user_id
variable SVERWEIS not constant within user_id
variable minalpha not constant within user_id
variable maxalpha not constant within user_id
variable mindelta not constant within user_id
variable maxdelta not constant within user_id
Your data are currently long. You are performing a reshape wide. You typed something like

. reshape wide a b, i(user_id) j(method_nr)

There are variables other than a, b, user_id, method_nr in your data. They must be constant within user_id because that is
the only way they can fit into wide data without loss of information.

The variable or variables listed above are not constant within user_id. Perhaps the values are in error. Type reshape error
for a list of the problem observations.

Either that, or the values vary because they should vary, in which case you must either add the variables to the list of xij
variables to be reshaped, or drop them.
r(9);

Now I tried your code and I get this:

encode method, gen(m)

. ssc install fre, replace
connection timed out -- see help r(2) for troubleshooting
http://fmwww.bc.edu/repec/bocode/f/ either
1) is not a valid URL, or
2) could not be contacted, or
3) is not a Stata download site (has no stata.toc file).
r(2);

Tony, please refer to Felix's solution in #11. Your comparison doesn't need other variables, and you may drop them first. With a data only including user_id, method_nr, mean_alpha, and mean_delta, you may run the codes in #7.

[Tony Josipovic]

Tony, please refer to Felix's solution in #11. Your comparison doesn't need other variables, and you may drop them first. With a data only including user_id, method_nr, mean_alpha, and mean_delta, you may run the codes in #7.

I conducted the code as Felix wrote in #11:
import excel using Data, firstrow clear
destring, replace
encode method, gen(m)
tab m
keep m mean_alpha mean_delta user_id
reshape wide mean_alpha mean_delta, i(user_id) j(m)

The results are not insignificant.
//ttest mean_alpha
ttest mean_alpha1 == mean_alpha2 // p=0.0002
ttest mean_alpha1 == mean_alpha3 // p=0.0002
ttest mean_alpha1 == mean_alpha4 // p=0.1279
ttest mean_alpha1 == mean_alpha5 // p=0.0019
ttest mean_alpha1 == mean_alpha6 // p=0.0366

ttest mean_alpha2 == mean_alpha3 // p=0.6262
ttest mean_alpha2 == mean_alpha4 // p=0.1557
ttest mean_alpha2 == mean_alpha5 // p=0.0000
ttest mean_alpha2 == mean_alpha6 // p=0.2151

ttest mean_alpha3 == mean_alpha4 // p=0.1594
ttest mean_alpha3 == mean_alpha5 // p=0.0000
ttest mean_alpha3 == mean_alpha6 // p=0.1107

ttest mean_alpha4 == mean_alpha5 // p=0.0000
ttest mean_alpha4 == mean_alpha6 // p=0.1440

ttest mean_alpha5 == mean_alpha6 // p=0.0000


//ttest mean_delta
ttest mean_delta1 == mean_delta2 // p=0.0000
ttest mean_delta1 == mean_delta3 // p=0.0440
ttest mean_delta1 == mean_delta4 // p=0.0000

ttest mean_delta2 == mean_delta3 // p=0.0025
ttest mean_delta2 == mean_delta4 // p=0.0000

ttest mean_delta3 == mean_delta4 // p=0.0000


Does anyone have another idea how else to analyze the data?

I have also conducted the same code wit data I got one month later. Here I get the folowwing error:

. reshape wide mean_alpha mean_delta, i(user_id) j(m)
variable m contains missing values
r(498);


Best

[Fei Wang]

Tony, your code seems correct, and results do show that some pair of measures have significant difference -- that would be your conclusions, no matter whether they meet your expectation.

[Tony Josipovic]

Tony, your code seems correct, and results do show that some pair of measures have significant difference -- that would be your conclusions, no matter whether they meet your expectation.

I see...any suggestions about some apropriate graphs to visualize the data?

Or any idea, about this error:

. reshape wide mean_alpha mean_delta, i(user_id) j(m)
variable m contains missing values
r(498);

Thanks.