Foreach loop - how to skip no observation - can not get 'capture' working for me

[Tobias Blessing]

Hello everyone,

I am kind of new to Stata and have been reading in this forum a lot. I learned a lot from it so far, but now it seems like I am stuck.

Stataversion: Stata SE 14.2

Here is my issue:

I am trying to build a dynmiac new regression for a sample of companies in a semi annually interval. Thus, I need to run a regression, identify the lags for newey and then use newey with the correct amount of lags. I build a loop and so far it worked out perfectly for complete datasets - but as soon as one company is lacking observations for a timeperiod, it breaks the loop with 'r(2000) no observations'.




Here is how my data is structured (excerpt) :

ID	year	key	fx	u1	u2	u3
1	2001/1	1	0.1	0.1	.	0.2
2	2001/1	1	0.2	-0.3	.	-0.1
3	2001/1	1	0.1	0.2	.	0.2
...	...	...	...	...	...	...
131	2001/2	1	0.3	0.5	0.1	0.3
...	...	...	...	...	...	...
2000	2008/2	16	0.2	-0.1	0.3	-0.2

This is the code I am currently using:

local i=0

foreach a of numlist 1/16 {
    local i=`i'+1
    
            foreach element of varlist u* {
            
                regress `element' fx if key==`i'
                estat durbinalt, lags(1/3)
                matrix A = r(p)
                    quietly scalar A1 = A[1,1]
                    quietly scalar A2 = A[1,2]
                    quietly scalar A3 = A[1,3]
                    quietly scalar B = 0.05
                    
                    quietly if(A1>B) {
                            scalar C = 0
                        }
                        quietly if(A1<B) & (A2>B) {
                            scalar C = 1
                        }
                        quietly if(A1<B) & (A2<B) & (A3>B) {
                            scalar C = 2
                        }

                    
                    newey `element' fx, lag(`=C'), if key==`i'

                                matrix test= nullmat(test) \ _b[FX]
                                local regressanden `regressanden'
                    }
    }

matrix colnames test = gamma
matrix rownames test = `regressanden'
matrix list test

The code breaks with no observations for 2001/1 in u2.

I already tried to work with capture and read a lot of topics and discussions & the stata help tutorials, but somehow I can not fit in capture correctly it seems: I tried to put it before the inner loop: capture foreach element of varlist u* {. This makes the code work, but although there are observations for 2001/1 & u3 it just skips them and continues listing the values for 2001/2 & u1. Thus the output for 2001/1 only includes the newey regression for u1.b Since in 2001/2 there are observations for u2 as well, it does not break here and lists values for u1, u2 and u3 in 2001/2.

Please note, I am new to stata, thus I might have some misunderstanding regarding the capture command here.

Can anyone relate to this and help me out here?

Thank you so much in advance

[Nick Cox]

First note that your code can be simplified in various ways. I hope that this is a faithful reduction. I presume that the local regressanden is defined upstream of this, but either way you can take it outside the loop.

local B = 0.05 
local regressanden `regressanden'

forval i = 1/16 {
    foreach element of varlist u* {
        regress `element' fx if key==`i'
        estat durbinalt, lags(1/3)
        matrix A = r(p)
        scalar A1 = A[1,1]
        scalar A2 = A[1,2]
        scalar A3 = A[1,3]
  
        if A1 > `B' local C = 0
        else if A1 < `B' & A2 > `B' local C = 1
        else if A1 < `B' & A2 < `B' & A3 < `B' local C = 2
                    
        newey `element' fx if key==`i' , lag(`C') 
        matrix test= nullmat(test) \ _b[FX]
     
  }
}

matrix colnames test = gamma
matrix rownames test = `regressanden'
matrix list test

I have to guess too at where to put the capture but consider this:

local B = 0.05 
local regressanden `regressanden'

forval i = 1/16 {
    foreach element of varlist u* {
        capture regress `element' fx if key==`i'

        if _rc == 0 { 
            estat durbinalt, lags(1/3)
            matrix A = r(p)
            scalar A1 = A[1,1]
            scalar A2 = A[1,2]
            scalar A3 = A[1,3]
  
            if A1 > `B' local C = 0
            else if A1 < `B' & A2 > `B' local C = 1
            else if A1 < `B' & A2 < `B' & A3 < `B' local C = 2
                    
            newey `element' fx if key==`i' , lag(`C') 
            matrix test= nullmat(test) \ _b[FX]
        } 
        else matrix test = nullmat(test) \ . 
  }
}

matrix colnames test = gamma
matrix rownames test = `regressanden'
matrix list test

[Clyde Schechter]

You need to -capture- only the -reg- command. And then you need to check that the problem was just no observations (code 2000) or insufficient observations (code 2001). So the inner loop would look like this:

foreach element of varlist u* {

    capture regress `element' fx if key==`i'
    if c(rc) == 0 {    // EVERTHING IS OK, PROCEED
        estat durbinalt, lags(1/3)
        matrix A = r(p)
        quietly scalar A1 = A[1,1]
        quietly scalar A2 = A[1,2]
        quietly scalar A3 = A[1,3]
        quietly scalar B = 0.05
        
        quietly if(A1>B) {
                scalar C = 0
        }
        quietly if(A1<B) & (A2>B) {
            scalar C = 1
        }
        quietly if(A1<B) & (A2<B) & (A3>B) {
            scalar C = 2
        }

        
        newey `element' fx, lag(`=C'), if key==`i'

        matrix test= nullmat(test) \ _b[FX]
        local regressanden `regressanden'
    }
    else if !inlist(c(rc), 2000, 2001) {    // UNEXPECTED ERROR
        display as error "Unexpected error in regression"
        exit c(rc)
    }
}

Note: Not tested. Beware of typos, etc.

Added: Crossed with #2 which is essentially the same solution, though that one does not check for and break on unanticipated errors.

[Tobias Blessing]

Thank you Nick and Clyde, both responses really helped me out.

I used Nick's improvements for my code and your code to find the correct place to put the capture.

My code works & looks like this:

local B = 0.05  
local regressanden `regressanden'  

forval i = 1/16 {
      foreach element of varlist u* {        
        capture {        
        regress `element' fx if key==`i'        
        estat durbinalt, lags(1/3)
      
        matrix A = r(p)        
        scalar A1 = A[1,1]        
        scalar A2 = A[1,2]        
        scalar A3 = A[1,3]            
        if A1 > `B' local C = 0
        else if A1 < `B' & A2 > `B' local C = 1        
        else if A1 < `B' & A2 < `B' & A3 < `B' local C = 2                              
      
       newey `element' fx if key==`i' , lag(`C')          

       matrix test= nullmat(test) \ _b[FX]    
       }  
    }
}  
matrix colnames test = gamma
matrix rownames te

Topic can be closed, thanks for your fast answers I really appreciate it!

[Clyde Schechter]

No, you didn't use my code to place the capture. What you did is different and, in my view, ill-advised. You -capture-d everything from -regress- through -matrix test-. And you did not check what the captured return code is. So if any error arises anywhere inside that loop, that value of key will be skipped over without any warning/notification and you will not know why. (And if you don't scrutinize your output you may not even realize that it happened.)

It is common in this kind of situation to have groups that turn out to have too few observations to include in the desired regression, and, from what I have observed here on Statalist, it seems that people prefer to simply skip over those groups. Fair enough. The code I posted in #3 will do that. But you are going much farther: you are asking Stata to ignore anything that goes wrong in any of those commands and not even tell you about it, let alone do anything about it. I think that's an unsafe programming practice and I recommend you reconsider it. Groups that have insufficient data are an anticipated problem and evidently considered not worth troubling ourselves over. But you are not on terra firma in presuming that nothing else can go wrong there.