using bys command inside a loop (or not)

[Sanghamitra Bandyopadhyay]

Dear Colleagues


I need to run a loop on a program which generates probabilities of 'spells' . The program inside the loop generates the probability of spells for one individual,and uses the bys command to generate the length of the spells and the the probabilities. I need to run a loop on it for a full dataset of individuals, where each individual has a person identification number (pid).

Below is my code. My understanding is that one cannot use the bys command in the loop, the program thus works very well without a loop, for one indiviudual, but doesn't within a loop. However, I need the bys command to be able to ask Stata to count the length of the spell for each group (hence the bys group).

Please can someone assist me with getting this code to run.


su pid, meanonly
forval i = 1/`r(max)' {
gen group= x != x[_n-1] if pid == `i'
replace group = sum(group) if pid == `i'
bys group: gen lengthspell =_N if pid == `i'
gen p_hat_spell = lengthspell/nyear if pid == `i'
}

Many thanks, Sanghamitra

[Romalpa Akzo]

Try this one, Sanghamitra. Good luck.

su pid, meanonly 
forval i = 1/`r(max)' {
gen group`i'= x != x[_n-1] if pid == `i'
replace group`i' = sum(group`i') if pid == `i'
bys group`i': gen lengthspell`i' =_N if pid == `i'
gen p_hat_spell`i' = lengthspell`i'/nyear if pid == `i'
}

[Nick Cox]

I don't see that you need any loop here. Consider as a start (inserting whatever defines order in panels in place of time)

bysort pid (time) : gen group = sum(x != x[_n-1]) 
bys pid group: gen lengthspell =_N

Also tsspell (SSC) and http://www.stata-journal.com/sjpdf.h...iclenum=dm0029

[Romalpa Akzo]

The loop might be needed if the issue is an extension of the original one (which is also raised by Sanghamitra) at https://www.statalist.org/forums/for...n-one-variable.

If it is the case, the original order must be kept to get the longest spell for each value of pid. My suggestion at #2 just serves for making the loop runable.

[Nick Cox]

Romalpa: Absolutely: your code in #2 has the considerable merit of making illegal code legal.

My small point goes beyond that. It is often easier to modify code than to think up a good solution (especially without any data example!).

The longest spell for each panel is yet another application of by:

bysort pid (lengthspell) : gen longest = lengthspell == lengthspell[_N]

Note that if there are ties for longest, all of them are tagged with 1.

[Sanghamitra Bandyopadhyay]

Dear Romalpa, Nick, thank you both for your codes! I think I tried a version of Romalpa's code before and it didn't work, but may be I got it wrong. I will try both codes and let you know. Thank you again for your prompt responses! best wishes, Sanghamitra

[Sanghamitra Bandyopadhyay]

Dear Romalpa, Nick, Many thanks for your help. I had similar problems with Romalpa's recommended code (as I had when I tried to run it earlier myself). However, Nick's code has worked very well, thank you to you both!

After this step, I have calculated the probabilities and I need to collapse the distribution to obtain just the individual spells and their associated probabilities, (so spell 1 has a prob p1, spell2 has a probability p2 .. etc), so that I can then estimate my index (which is a function of the spells and their associated probabilities). For an invididual pid, I just run

collapse (mean) p_hat_spell, by(group)

But for all pids when I do:
bys pid group: collapse (mean) p_hat_spell,

it doesn't generate the spell distribution as one cannot use collapse with bys.
Can you so kindly point towards what commands I can use to generate the collapsed spell distribution, thank you so much again!

Best wishes, Sanghamitra

[Romalpa Akzo]

I am not very sure that I understand correctly your issue. But have you ever just tried:

collapse (mean) p_hat_spell, by(pid group)

[Sanghamitra Bandyopadhyay]

Yes! It has worked, thank you so much for getting back so quickly! Best wishes and have a good day!