clttest gives me different results every time I run it

[Josue Ortega]

Hi, I am running something like

clttest truth, cluster(userid) by(envyfree)

and every time I run the command I get different results, which is super shady. Truth and envyfree are 0-1 variables. Any possible explanations would be most helpful.

[Leonardo Guizzetti]

Can you post a reproducible data example that shows what you claim? It's been a while since I've used this but I don't recall this behaviour when I did.

[William Lisowski]

I don't have any knowledge of clttest, but here's an example of the problem. Perhaps Jeph Herrin - the author of this community-contributed command available through SSC - will see this and comment.

. sysuse auto, clear
(1978 automobile data)

. drop if missing(rep78)
(5 observations deleted)

. replace price = price/1000
variable price was int now float
(69 real changes made)

. clttest price , cluster(rep78) by(foreign)

 t-test adjusted for clustering
 price by foreign, clustered by rep78
 ------------------------------------------------------------------------
  Intra-cluster correlation         =           0.0000
 ------------------------------------------------------------------------
              N    Clusts    Mean           SE             95 % CI
 foreign=0    48     5       6.1792      0.4175       [  5.0201,  7.3384]
 foreign=1    21     3       6.0701      0.6312       [  3.3543,  8.7859]
 ------------------------------------------------------------------------
 Combined     69     3       6.1460      0.3482       [  5.2940,  6.9981]
 ------------------------------------------------------------------------
 Diff(0-1)    69     8       0.1091      0.7568       [ -1.7427,  1.9609]

 Degrees freedom:    6

                    Ho: mean(-) = mean(diff) = 0

  Ha: mean(diff) < 0         Ha: mean(diff) ~= 0        Ha: mean(diff) > 0
       t =   0.1442                t =   0.1442              t =   0.1442
   P < t =   0.5550          P > |t| =   0.8901          P > t =   0.4450

. clttest price , cluster(rep78) by(foreign)

 t-test adjusted for clustering
 price by foreign, clustered by rep78
 ------------------------------------------------------------------------
  Intra-cluster correlation         =           0.0000
 ------------------------------------------------------------------------
              N    Clusts    Mean           SE             95 % CI
 foreign=0    48     5       6.1792      0.4132       [  5.0319,  7.3266]
 foreign=1    21     3       6.0701      0.6248       [  3.3820,  8.7583]
 ------------------------------------------------------------------------
 Combined     69     3       6.1460      0.3447       [  5.3027,  6.9894]
 ------------------------------------------------------------------------
 Diff(0-1)    69     8       0.1091      0.7491       [ -1.7238,  1.9420]

 Degrees freedom:    6

                    Ho: mean(-) = mean(diff) = 0

  Ha: mean(diff) < 0         Ha: mean(diff) ~= 0        Ha: mean(diff) > 0
       t =   0.1457                t =   0.1457              t =   0.1457
   P < t =   0.5555          P > |t| =   0.8890          P > t =   0.4445

.

[Josue Ortega]

I got this email from Jeph, but don't really get it. I will think about it.

From: Jeph Herrin <[email protected]>
Sent: 09 September 2021 14:24
To: Josue Ortega <[email protected]>
Subject: Re: Clttest gives different results every time is run


This happens when your -cluster-s are not nested within your -by- group. If some clusters have two different values of -envyfree-, you will get different answers each time.

Hope this helps.

cheers,
Jeph

[Leonardo Guizzetti]

If you tabulate cluster I'd with your by group variable, you should find that each cluster fits entirely into exactly one by-variable valuable.

What does this output?

tab userid envyfree

[Josue Ortega]

Something like this Leonardo:


| envyfree
id | 0 1 | Total
-----------+----------------------+----------
1 | 21 21 | 42
2 | 21 21 | 42
3 | 21 21 | 42
4 | 21 21 | 42

[Leonardo Guizzetti]

Very well. That is being interpreted by -clttest- such that every cluster has members in each group of envyfree. You will need to make cluster ids unique. For example, this would assign a unique integer to each cluster. Then use uniqueid as your id.

egen uniqueid = group(userid envyfree)

[Josue Ortega]

Thanks a lot Leonardo, most helpful.

[Jeph Herrin]

Thanks to Kit Baum, an updated version of the package -cltest- is now available on SSC. The updated version now checks the assumption described above (that clusters are nested within -by- groups), and refuses to continue if the assumption is not met.

cheers,
Jeph

[Jahan Ismat]

Hello stata experts,

I was trying to use clttest command. Whenever I try to run the following command,

clttest price , cluster(rep78) by(foreign)

I get this error message

Clusters must be nested within by() variable

Can anyone please help?

Best regards,
Jahan