Forvalues loop using if and preserve / restore, but cannot save files

[Jaepil Lee]

Dear all,

First of all, happy new year!

I am writing to ask a question about using a loop with `if', combined with preserve/restore.
The following posts are what I have read so far:

https://www.stata.com/statalist/arch.../msg01117.html
https://www.stata.com/statalist/arch.../msg00105.html
https://www.stata.com/statalist/arch.../msg00484.html
https://www.statalist.org/forums/for...d-saving-files
https://www.stata.com/support/faqs/p...-if-qualifier/

I have a program defined as `match', which requires one argument:

capture program drop match
program define match
keep if ~~~
by var1 : ~~~
end

Using the program above, I want to do a loop using forvalues:

forvalues t=100(1)200{
if t==100 {
preserve
match `t'
save data.dta, replace
restore
}
else {
preserve
match `t'
append using data.dta
save data.dta, replace
restore
}
}

If I run this code, however, I get `data.dta not found' and the loop stops right away.
This loop works only if data.dta is already saved: Thus, I am running the loop after I saved an empty dta file using a command below:
save data.dta, replace emptyok
Although I got it solved, but I want to know why it needs an empty dta file first; should not `replace' option enough to ensure save command to run, if there is no data.dta file?

[Clyde Schechter]

This does seem a bit strange to me. To really diagnose what's going on, you should -set tracedepth 1- and -set trace on- at the top of this code so that you will be able to see which command is complaining, rather than just knowing that it was somewhere inside a loop.

That said, I'll make a guess. -save- will not save an empty file unless the -emptyok- option is specified. My guess is that on the first run through the loop, your program match leaves you with an empty data set, which the first round of the loop fails to save. Then when you get to the second round which uses -append-, there is no file data.dta and Stata complains. The weakness of this theory is that you should have actually seen Stata quit with "no variables defined" at the first -save- command. So this theory may be wrong.

[Joro Kolev]

I would second Clyde, the problem is somewhere else, not where you think. If you do not have a file with the given name, and you specify replace, Stata does:

. save nlshort, replace
(note: file nlshort.dta not found)
file nlshort.dta saved

[David Benson]

EDIT: This crossed with Joro's post - so it's probably not that you need to remove ", replace"
I wondered if Stata didn't like the

save data.dta, replace

the first time around (so could try removing the "replace" in the if t==100 loop and see if it worked correctly after that). But, that is just a guess.

[Joro Kolev]

However, in all likelihood this loop does not do what you think that it is doing, the problem (I think) is in the statement:

if t==100 {

This code is not referring to the macro t that is running from 100 to 200.

What happens here (most probably) is that you have a variable in your data whose name starts with "t", and the code above is referring to the first observation of this variable.

[Joro Kolev]

Here some examples:

. sysuse auto, clear
(1978 Automobile Data)

. keep in 1/2
(72 observations deleted)

. list price

     +-------+
     | price |
     |-------|
  1. | 4,099 |
  2. | 4,749 |
     +-------+

. if price==4099 dis "Yes indeed, this is the first observation"
Yes indeed, this is the first observation

. if price==4749 dis "Yes indeed, this is the first observation"

. if p==4099 dis "Yes indeed, this is the first observation"
Yes indeed, this is the first observation

. if p==4749 dis "Yes indeed, this is the first observation"

[Jaepil Lee]

Thank you all for your sincere answers.

Following what Clyde suggested, I have found below:

- forvalues t=100(1)200{
- if t==100{
preserve
match `t'
save data.dta, replace
restore
}
- else{
- preserve
- match `t'
= match 100
------------------------------------------ begin match ---
(program commands working smoothly)
-------------------------------------------- end match ---
- append using data.dta
------------------------------------------------ begin append ---
- version 11
- if (_caller() < 11) {
local version : di "version " string(_caller()) ":"
`version' _append `0'
exit
}
- syntax [anything(everything)] [, GENerate(name) * ]
- gettoken using filenames : anything
- if (`"`using'"' != "using") {
= if (`"using"' != "using") {
di as err "using required"
exit 100
}
- if (`"`filenames'"' == "") {
= if (`" data.dta"' == "") {
di as err "invalid file specification"
exit 198
}
- foreach filename of local filenames {
- capture quietly describe using `"`filename'"'
= capture quietly describe using `"data.dta"'
- if (_rc) {
- di as err `"file `filename' not found"'
= di as err `"file data.dta not found"'
file data.dta not found
- exit 601
}
}
-------------------------------------------------- end append ---
save dataLL.dta, replace
restore
}
}

So, it seems like regardless of `t' value I am using in the forvalues loop, the logic falls into `else' statement.

As Joro said, t==100 was referring to the variable t, not the argument of the loop. Since I did not have t variable in the data, the loop always fell into `else' statement (in this case, t!=100) and tried to do append instead.

To fix this, I changed t==100 to `t'==100. <--- This is how we refer to an argument for a loop
save -filename-, replace was okay after all.

This was my first post to Statalist. Just like how much I found other posts in this forum helpful for my Stata coding, I cannot thank you all enough.

[Clyde Schechter]

Not to put too fine a point on it, but

As Joro said, t==100 was referring to the variable t, not the argument of the loop. Since I did not have t variable in the data, the loop always fell into `else' statement (in this case, t!=100) and tried to do append instead.

it is not correct to say "I did not have t variable in the data," because if that were true, the -if t == 100- command would have itself triggered an error message. Remember that, unless you set variable abbreviation off, if you specify the start of a unique variable name, Stata will fill in the rest of it. So I conclude that you do in fact have in your data set one, and only one, variable that begins with t. And it is that variable, specifically the first observation of that variable, that was being compared to 100.

[Joro Kolev]

I am glad that you made it all work ! However two remarks:

1. "To fix this, I changed t==100 to `t'==100. <--- This is how we refer to an argument for a loop"
The `macro_name_here' is called dereferencing a macro, in this case a local. We always dereference macros like this, whether they an argument in a loop, or not.

2. (I believe) you have to have something in the data that starts with -t-. Either a variable, or scalar. If there is nothing starting with -t- Stata should generate an error like this:

. sysuse auto, clear
(1978 Automobile Data)

. keep price

. if f==100 dis "Say something"
f not found
r(111);

And the same thing happens if there is an else statement. To running the code:

sysuse auto, clear

keep price

if f==100 {
dis "Something we should not see"
}

else {
dis "Can we even reach here?"
}

Stata returns upon execution:

. do "C:\Users\Gueorgui1\AppData\Local\Temp\STD1374_000000.tmp"

. sysuse auto, clear
(1978 Automobile Data)

. 
. keep price

. 
. if f==100 {
f not found
r(111);

end of do-file

r(111);

[Joro Kolev]

While I was generating examples to try and persuade you that if you do not have a variable starting with -t-, you will never reach the -else- part of the statement, Clyde explained a lot better than me why this is so

[Jaepil Lee]

Not to put too fine a point on it, but


it is not correct to say "I did not have t variable in the data," because if that were true, the -if t == 100- command would have itself triggered an error message. Remember that, unless you set variable abbreviation off, if you specify the start of a unique variable name, Stata will fill in the rest of it. So I conclude that you do in fact have in your data set one, and only one, variable that begins with t. And it is that variable, specifically the first observation of that variable, that was being compared to 100.

Indeed, as Joro and Clyde pointed out, I DO have a variable named `time'. So, it must be that the loop compared the first observation of time variable, which is different from 100. Thank you!