Experience with pc_simulate- Burlig et al (2020)

[Lanna Ng]

Hi,

I am designing a RCTs with 36 clusters and 2 treatment arms. Experienced researchers criticized that my study is severely under-powered, and I should recruit more units. However, it is not easy due to resources constraint.

Lately I have found Stata module called pcpanel introduced by Burlig et al (2020). I used pc_simulate function to compute the power. Since I already have baseline data and reliable information about serial correlation of outcomes variable, I was able to simulate the 1st and 2nd post-treatment period. My code is here:

set seed 123
gen u1= rnormal(0,1)
gen u2= rnormal(0,1)
gen outcome_t2= outcome_t1*0.23+u1
gen fgcount_t3=fgcount_t2*0.15+u2

reshape long outcome_t, i (hhid) j(time)

pc_simulate outcome_t, model(ANCOVA) mde(0.24) i(hhid) t(time) pre(1) post(1 2) idcluster(villageid) p(0.5) vce(cluster villageid) control(age education) bootstrap

Then the results I received is so surreal !! It proves that my design is perfect and no need to change. It is 30% higher than sampsi or power commands.

However, because it is too good to be true, I wonder if anyone here has experience with pc_simulate. Is it really reliable and it is any risk I was unconsciously felt into?

Thank you so much.

Lanna

[Bert Lloyd]

I may be missing some context but
1 - it may be too optimistic to have normally distributed errors
2 - if I'm understanding your code correctly, you do not have any cluster-level shocks to the outcome variable, and that's what will hurt power

[Lanna Ng]

Thanks Bert,

They did not have option to do cluster-level shock in fact. The program accounted to outcome fixed effects already from group variables.

However, when I try to compute individual randomization, it produces less power than cluster design one. It is completely contradict to the rule of thumb, power increases when number of unit increases. It is so strange. Here is the code I ran for individual level:

 gen u1= rnormal(0,1)
gen u2= rnormal(0,1)
gen outcome_t2= outcome_t1*0.23+u1
gen fgcount_t3=fgcount_t2*0.15+u2  
reshape long outcome_t, i (hhid) j(time)
pc_simulate fgcount_t, model(ANCOVA) mde(0.24) i(index) t(time) pre(1) post(1 2) p(0.5) control(b1q3 b1q2) stratify(a2q3) replace

I do not know what happened