Append to a file on disk on each iteration of a loop

[John Larsson]

Hello,

I am using Stata version 15.1 and would like to create a file on disk and append the contents of the current dataset in memory to the file on each iteration of a loop:

The scenario is as follows:

levelsof catVar, local(levels)
foreach l of local levels {
    preserve
    drop if catVar != `l'

    // here I wish to create some new variables
    // based on the values of the current variables 
    // in memory, and append to a file on disk.
    
    restore
}

The Stata append command appears to be only for appending from disk to the current dataset while I wish to append in the other direction.

If there is no way of doing this I would appreciate any alternative suggestions, such as iterating overall all current rows in the dataset and writing them out one-by-one.

[Clyde Schechter]

You can do it, but you will really be thrashing the disk. Your grandchildren will no doubt appreciate seeing the results when they're done. Better is to just be sure that you retain the value of catVar in the data set you are building up in each iteration. Then when you are done, sort the final data set on catVar in whichever direction you prefer. (See -help gsort-).

[John Larsson]

Hi Clyde,
Yes, I'd prefer a more efficient approach. The data is already sorted by catVar and time (quarter).
The problem is that for each observation, I want to create a new variable based on the values of the other observations corresponding to the same catVar value.
Dropping the other observations allows me to index into the first instance of interest, as in

 Beta[1]

.
Basically, for each value of catVar, I need to loop sequentially over all the instances corresponding to that catVar value.

[Clyde Schechter]

I do not understand, at this point, where you are and where you want to go. From #3 it sounds like you have a single data set available that includes all the data you need for your analyses and the problem you face is how to do a calculation repeatedly among groups of observations characterized by a distinct value of catVar. The most efficient way to do that is with the -runby- command, written by Robert Picard and me, and available from SSC.

You don't provide any details about what these calculations are, so I can't be more specific than that. Fortunately, the help file for -runby- contains many kinds of worked examples illustrating its use.