Help with loop

[Sunita Rao]

I have 1200 CIQ_ID, each of which is a company. For each CIQ_ID/company, I have raw returns and market returns.

I want perform 1200 regressions as follows: regress raw_returns MktReturns. Further, I want to to store residuals for each regression and the calculate standard deviations for each set of the residuals (i.e. for each CIQ_ID/company).

I use the following code, but I am getting an error:

levelsof CIQ_ID, local(levels)

gen residual=.

foreach i of local levels {
regress raw_returns MktReturns if CIQ_ID==`i'
predict temp, residuals
replace residual=temp if CIQ_ID==`i'
drop temp
}

egen idio_volatility=sd(residual), by(CIQ_ID)
order idio_volatility, after(stock_volatility)




foreach i of local levels {
2. regress raw_returns MktReturns if CIQ_ID==`i'
3. predict temp, residuals
4. replace residual=temp if CIQ_ID==`i'
5. drop temp
6. }
IQ100231 not found
r(111);

Here is a sample of my data:

* Example generated by -dataex-. For more info, type help dataex
clear
input str12 CIQ_ID float(raw_returns MktReturns)
"IQ100231"   .007290433  .0026284044
"IQ100231"  -.005665738   .002994473
"IQ100231"   .007278641            .
"IQ100231" -.0016129036            .
"IQ100231"  .0036253815   -.00620371
"IQ100231" -.0036253815  -.010129768
"IQ100231"  -.012588998  -.017621646
"IQ100231"  -.011508553   .011453757
"IQ100231"  -.017094433   .010341487
"IQ100231"  -.002947991   .008782399
"IQ100231"   -.06715687  -.016217556
"IQ100231"   .005398124   .007499604
"IQ100231"   .006706933   .009178631
"IQ100231"   .017668305  -.009503838
"IQ100231"   .004368727   .007205737
"IQ100231"   .009976227  -.003081058
"IQ100231"  -.003891897  -.020494226
"IQ100231"   .008628181  -.011663516
"IQ100231"    .00770222   .008485059
"IQ100231"  -.017631074  -.014095594
"IQ100231"  -.036223043    .01188673
"IQ100231"    .02663273    .01914309
"IQ100231"     .0453845  -.013114273
"IQ100231"   .013718776            .
"IQ100231"    .01026914            .
"IQ100231"    .01178157 -.0022927218
"IQ100231"  -.002426204 -.0016544486
"IQ100231"  -.023347365  .0032003694
"IQ100231"   .010717332  .0014010826
"IQ100231"   .011009285 -.0033793035
"IQ100231"            0  .0026042596
"IQ100231"   .005661156  -.003856391
"IQ100231"   .005629287 9.364482e-06
"IQ100231" -.0008022463  -.007196296
"IQ100231"  .0012031283 -.0005511063
"IQ100231"  -.009665802   .008260587
"IQ100231"   .006856248   .003505053
"IQ100231"   .008803578 -.0008884214
"IQ100231"  -.012024193 -.0037261166
"IQ100231"   .015206375 -.0041723736
"IQ100231"   .001983734  -.006439696
"IQ100231"   .005533611  -.003673685
"IQ100231"  -.004345257   -.00179983
"IQ100231"     .0114152 -.0019477884
"IQ100231"  -.019364366  -.009780024
"IQ100231"   .012294424   .005063897
"IQ100231"   .010586258  -.001818312
"IQ100231"  -.005867414  -.004736889
"IQ100231"   .010536683  .0010738354
"IQ100231"   .005806094  -.002991252
"IQ100231"  -.012427344  -.003657657
"IQ100231"  -.010212186  -.007366186
"IQ100231"   .014501528  -.003369578
"IQ100231" -.0031176955  -.007431109
"IQ100231"    -.0125689  -.016919125
"IQ100231"   .005125184  -.003019938
"IQ100231"    .02332014   .002420735
"IQ100231"    .02803214   .006888909
"IQ100231"     .0118828  .0019167697
"IQ100231"  .0080882795  -.008265908
"IQ100231"  .0014635933 -.0040942547
"IQ100231"  -.003663008   -.01046882
"IQ100231" -.0033082184   .013070906
"IQ100231"  -.005538135  -.011425984
"IQ100231"   .014335846   .011981754
"IQ100231"  -.005856532 -.0015671804
"IQ100231"   .005856532   .020573413
"IQ100231"  -.009902887  .0027796596
"IQ100231"   .002209132  -.001458564
"IQ100231"   -.00664209  -.012063278
"IQ100231" -.0018528817  -.009474153
"IQ100231"   .009230266  .0008124529
"IQ100231"   .011691767   .017122941
"IQ100231"  .0025394536   .009153866
"IQ100231" -.0010875477  .0015520195
"IQ100231"  -.008743225   -.00843857
"IQ100231"   .006928013   .005762376
"IQ100231"  -.012431605 -.0022731265
"IQ100231"   .008791265   .007753606
"IQ100231"   .006543098  .0046149553
"IQ100231"  -.006543098  .0013192754
"IQ100231"   .005092772            .
"IQ100231"   .002898553            .
"IQ100231"  -.016414741  .0003336542
"IQ100231"   .008423416 -.0028371806
"IQ100231"  -.006953367 -.0003121891
"IQ100231"   -.00367918 -.0029475125
"IQ100231"  -.004062794  -.008769659
"IQ100231"   .015059972   .005842978
"IQ100231"   .000728863 -.0022169428
"IQ100231"   .002546845 -.0014953938
"IQ100231"   .001815212  -.008488887
"IQ100231"  -.001451906  -.008106836
"IQ100231"   .010838256 -.0012574414
"IQ100231"    .01285273    .01080644
"IQ100231"  .0003546728   .007086517
"IQ100231" -.0003546728  -.002616419
"IQ100231"  -.006048764 -.0016063297
"IQ100231"    .00498399  -.002947365
"IQ100231"   -.00248889 -.0024651375
end

I would be very grateful for some help with my code. I do not know what I am doing wrong.

[Clyde Schechter]

CIQ_ID is a string variable. So, everywhere in the code where you have

if CIQ_ID==`i'

it needs to be

if CIQ_ID==`"`i'"'

[Sunita Rao]

Dear Prof. Schechter,

Thank you very much for correcting me. This time the loops run fine. However, after I run the 1200 regressions Stata is giving me the following error:

(100,587 missing values generated)
(484 real changes made)
no observations
r(2000);

end of do-file

r(2000);

However, the variable residual has been created:

* Example generated by -dataex-. For more info, type help dataex
clear
input float residual
   .004279196
    -.0089022
            .
            .
   .006048123
  .0012128757
  -.003141347
   -.01994961
   -.02485116
  -.009745487
   -.05857309
 -.0006101299
 -.0003343457
    .02212146
 -.0014587244
    .01047775
   .007323117
    .01441008
  .0010876631
  -.010352834
   -.04493048
   .013460797
    .05205898
            .
            .
   .011798067
 -.0028024064
  -.026710505
   .008461208
   .011694304
  -.002996382
   .006639703
  .0042294203
  .0022311853
 .00014809216
  -.016142253
   .003305651
  .0079560755
  -.011125797
   .016379332
   .004551666
   .006399748
  -.004632013
   .011219475
   -.01474129
   .007784745
   .010310873
 -.0043471376
   .008481897
   .006252362
  -.011571067
  -.007074229
   .015180563
 .00006020549
 -.0035534764
   .005589102
    .02043667
    .02239962
   .009309396
   .011779793
   .002588487
   .001383851
  -.012744227
 .00009762195
    .00556994
  -.006286426
  -.008195411
  -.013007184
  .0017124113
 -.0006142374
  .0025820064
   .007336298
-.00023726645
 -.0044865888
  -.003436537
  -.004945481
  .0019885928
  -.012427164
   .002626736
  .0023096316
  -.008748891
            .
            .
  -.018014127
   .008774892
  -.008155398
  -.003259822
-.00006134711
   .010070962
  .0006987366
   .002072784
   .005643913
  .0021417376
   .010217794
   .004809938
  -.005399428
 -.0001390205
  -.006454572
   .005403258
 -.0023663144
end

The variable idio_volatility has not been created
The code that I ran is as follows:

levelsof CIQ_ID, local(levels)

gen residual=.

foreach i of local levels {
regress raw_returns MktReturns if CIQ_ID==`"`i'"'
predict temp, residuals
replace residual=temp if CIQ_ID==`"`i'"'
drop temp
}

egen idio_volatility=sd(residual)
order idio_volatility, after(stock_volatility)



If I run the last 2 lines again, the variable idio-volatility is just one single value for all my observations.


I would be grateful for some advice. I do not know how to proceed.

[Clyde Schechter]

OK. The "no observations" message means that there is a value (perhaps more than one) of CIQ_ID where, due to missing values of MktReturns, there are no observations to use in the regression. Remember that only observations that have non-missing values of both raw_returns and MktReturns, can be used in the regression. And a regression with one explanatory variable requires a minimum of 2 observations. So, there must be some value of CIQ_ID where every observation has a missing value for one of those two variables. This is a common problem with this kind of analysis, and it comes up frequently on Statalist. Here's what I recommend you do:

levelsof CIQ_ID, local(levels)

gen residual=.

foreach i of local levels {
    capture regress raw_returns MktReturns if CIQ_ID ==`"`i'"'
    if c(rc) == 0 {
        predict temp, residuals
        replace residual=temp if CIQ_ID ==`"`i'"'
        drop temp
    }
    else if inlist(c(rc), 2000, 2001) {
        continue
    }
    else {
        display in red `"Unexpected error with CIQ_ID == `"`i'"'"'
        exit c(rc)
    }
}

This code will calculate the residuals when the regression has been successful. If the regression fails because there are no observations, or only one observation ("insufficient observations"), the Stata will proceed to the next value of CIQ_ID and issue no error messages. In the event that the -regress- command produces some other error, Stata will stop with an error message. If you wish to, (and I think you should) you can then go back and look at the CIQ_ID's with failed regressions by running:

levelsof CIQ_ID if missing(residual)

You can then check the data on those observations to see if the missing values represent data errors (and, if so, you should fix them and start over) or are just unfortunate.

The variable idio_volatility is not created because when Stata encounters an error like the one you got, it halts execution. So that command was never reached: execution aborted during the loop. With the code shown in this post, you will not have that problem unless you encounter an unanticipated error (something other than no observations or insufficient observations).

As for the command creating only a single value of idio_volatility, that is happening because that is what you told Stata to do. I'm guessing that you intended to have Stata create a separate value for each CIQ_ID. If so, the command should be:

by CIQ_ID, sort: egen idio_volatility = sd(residual)

Added: The code for the loop shown here suppresses the regression output. Since you have about 1200 regressions to run, the output would be voluminous and essentially impossible to work with anyway. But if you feel you need to see it, replace -capture- by -capture noisily-.

[Sunita Rao]

Dear Prof. Schechter,

I greatly appreciate your detailed explanation.

I did go back and check on failed regressions. They turn out to be unrecoverable.

This is a learning experience for me. Thank you again!