No Observation Error (Forvalues loop)

[Jade Li]

Dear All,

I am trying to run a loop after I have defined and called mata. I have several markets (n=17) and I would like to loop over the markets rather than just running the code for every market separately. I got some good help before with the mata code. But now after I run the code it gives me an error: no observations!

This is the code:

cd "$maindataset_jadPC"
use mdl_2021_first_part, clear


clear mata
mata:
void GMM_DLW(todo,betas,crit,g,H)
{
    PHI=st_data(.,("phi"))
    PHI_LAG=st_data(.,("phi_lag"))
    Z=st_data(.,("const","l_lag","k"))
    X=st_data(.,("const","l","k"))
    X_lag=st_data(.,("const","l_lag","k_lag"))
    Y=st_data(.,("y"))
    C=st_data(.,("const"))

    OMEGA=PHI-X*betas'
    OMEGA_lag=PHI_LAG-X_lag*betas'
    OMEGA_lag_pol=(C,OMEGA_lag)
    g_b = invsym(OMEGA_lag_pol'OMEGA_lag_pol)*OMEGA_lag_pol'OMEGA
    XI=OMEGA-OMEGA_lag_pol*g_b
    crit=(Z'XI)'(Z'XI)
}

void GMM_DLW_TL(todo,betas,crit,g,H)
{
    PHI=st_data(.,("phi"))
    PHI_LAG=st_data(.,("phi_lag"))
    Z=st_data(.,("const","l_lag","k","l_lag2","k2","l_lagk"))
    X=st_data(.,("const","l","k","l2","k2","lk"))
    X_lag=st_data(.,("const","l_lag","k_lag","l_lag2","k_lag2","l_lagk_lag"))
    Y=st_data(.,("y"))
    C=st_data(.,("const"))

    OMEGA=PHI-X*betas'
    OMEGA_lag=PHI_LAG-X_lag*betas'
    OMEGA_lag_pol=(C,OMEGA_lag)
    g_b = invsym(OMEGA_lag_pol'OMEGA_lag_pol)*OMEGA_lag_pol'OMEGA
    XI=OMEGA-OMEGA_lag_pol*g_b
    crit=(Z'XI)'(Z'XI)
}


 void DLW()
    {
S=optimize_init()
optimize_init_evaluator(S, &GMM_DLW())
optimize_init_evaluatortype(S,"d0")
optimize_init_technique(S, "nm")
optimize_init_nmsimplexdeltas(S, 0.1)
optimize_init_which(S,"min")
optimize_init_params(S,(1,0.8,0.2))
p=optimize(S)
p
st_matrix("beta_dlw",p)
}

void DLW_TRANSLOG()
    {
S=optimize_init()
optimize_init_evaluator(S, & GMM_DLW_TL())
optimize_init_evaluatortype(S,"d0")
optimize_init_technique(S, "nm")
optimize_init_nmsimplexdeltas(S, 0.1)
optimize_init_which(S,"min")
optimize_init_params(S,(0,0,0,0,0,0))
p=optimize(S)
p
st_matrix("beta_dlwtranslog",p)
}
end

//end mata here


cap program drop dlw
program dlw, rclass
preserve
sort ordinal year
mata DLW()
end
*------------------------------------------------------------------------------------------------*
cap program drop dlw_translog
program dlw_translog, rclass
preserve
sort ordinal year
mata DLW_TRANSLOG()
end



// use the loop here

forvalues j= 1/17  {
keep if markets ==`j'

//basic ols estimates
reg y l k i.year
gen beta_lols=_b[l]
gen beta_kols=_b[k]
gen Markup_ols=_b[l]/alpha_l

//ACF estimates
dlw
gen beta_c1=beta_dlw[1,1]
gen beta_l1=beta_dlw[1,2]
gen beta_k1=beta_dlw[1,3]

gen Markup_dlw1=beta_l1/alpha_l
gen omega_dlw1=phi-beta_l1*l-beta_k1*k

dlw_translog
gen betal_tl1=beta_dlwtranslog[1,2]
gen betal_tl2=beta_dlwtranslog[1,4]
gen betak_tl1=beta_dlwtranslog[1,3]
gen betak_tl2=beta_dlwtranslog[1,5]
gen betalk_tl=beta_dlwtranslog[1,6]

gen betal_tl=betal_tl1+2*betal_tl2*l+betalk_tl*k
gen mar_dlwz2=betal_tl/alpha_l

}

and this is the error:

Iteration 242: f(p) =   8423.563  
Iteration 243: f(p) =  8423.5613  
Iteration 244: f(p) =  8423.5613  
Iteration 245: f(p) =  8423.5612  
                  1              2              3              4              5              6
    +-------------------------------------------------------------------------------------------+
  1 |  -.0357106538    .0331540067     .022495456     .042301098   -.0053182375    .0396587528  |
    +-------------------------------------------------------------------------------------------+
(3 missing values generated)
(3 missing values generated)
(3 missing values generated)
(1,249 observations deleted)
no observations
r(2000);

end of do-file

r(2000);


How to cope with this error?
Thank you
.

I have also tried adding - capture -

// use the loop here

forvalues j= 1/17  {
keep if markets ==`j'

//basic ols estimates / trying with -capture-
capture reg y l k i.year
gen beta_lols=_b[l]
gen beta_kols=_b[k]
gen Markup_ols=_b[l]/alpha_l

//ACF estimates
dlw
gen beta_c1=beta_dlw[1,1]
gen beta_l1=beta_dlw[1,2]
gen beta_k1=beta_dlw[1,3]

gen Markup_dlw1=beta_l1/alpha_l
gen omega_dlw1=phi-beta_l1*l-beta_k1*k

dlw_translog
gen betal_tl1=beta_dlwtranslog[1,2]
gen betal_tl2=beta_dlwtranslog[1,4]
gen betak_tl1=beta_dlwtranslog[1,3]
gen betak_tl2=beta_dlwtranslog[1,5]
gen betalk_tl=beta_dlwtranslog[1,6]

gen betal_tl=betal_tl1+2*betal_tl2*l+betalk_tl*k
gen mar_dlwz2=betal_tl/alpha_l

}

the error is different now:

 23.
. }
(0 observations deleted)
variable beta_lols already defined
r(110);

end of do-file

r(110);

.

[William Lisowski]

After the first time through your loop, you only have observations for market 1 in memory - that is all that were retained by the keep command. The second time through the loop, there are no observations for market 2 in memory.

The best code would be to remove the current use command and replace

forvalues j= 1/17  {
keep if markets ==`j'

with

forvalues j= 1/17  {
use if markets ==`j' using mdl_2021_first_part, clear

which will just read in the observations you need for the current loop.

[Nick Cox]

This needs a cross-reference to https://www.statalist.org/forums/for...mata-forvalues

See general advice on bumping etc. at https://www.statalist.org/forums/help#adviceextras

Already we're making the same points as before!

[William Lisowski]

I will note that you may also find a fuller discussion of the issue in post #2 above in another topic: post #11 at

https://www.statalist.org/forums/for...ble-in-my-code

where someone is attempting something very much like what you are doing, including exactly the same mistake when looping over the data.

[Jade Li]

Dear William Lisowski,
Thank you so much for your answers!
I will check the links you have suggested too!

Dear Nick Cox,
Sorry for that, I did not pay attention to this thing.