faster to re-load data or preserve/restore?

[Jack Reimer]

I have a workflow that requires frequently collapsing data according to different specifications and then running regressions. The data are very large and typically take >15-30 seconds to do, what feels like, anything. In principle, is it computationally less expensive to use preserve/restore repeatedly or simply clear and re-load data sets every time?

I know that my workflow is probably more cumbersome than it needs to be and that the real answer is to bite the bullet and learn how to leverage mata but it would be helpful to know for future reference.

[Sergio Correia]

You probably won't see much difference unless you are using Stata 16, where you can preserve-restore using memory.

Besides that, to answer your question we would need to know a bit more on your data. A few misc tips:

1. If you can, trim the data as much as possible (only keep required obs and variables). Then, be sure that you are not using types that are too big for the dataset (a "double" variable actually takes 8 times the space of a "byte"). See "describe" for that, as well as "compress"
2. When you collapse, use the "fast" option!
3. Even better, use gcollapse (ssc install gtools)
4. It might also help if you use the collapse+append trick of fcollapse: fcollapse has an append option where the collapsed data is saved at the end of the dataset. So you can collapse by many things and not having to preserve/save (as long as you use the fast option)

Best
S

[Richard Williams]

To elaborate on Sergio's point, I wonder if Stata 16's new frame features would make your life easier. It makes it easy to have multiple data sets open at once. Stata Corps claims "The do- and ado-files that you have previously written that use preserve and restore will run faster if you use Stata/MP because it secretly uses frames in place of temporary files to preserve data. The speed-up is sometimes remarkable. We have old do- and ado-files that run 20 percent faster."

https://www.stata.com/new-in-stata/m...ets-in-memory/

[Eric Goldstein]

However, keep in mind that if the dataset is over a certain size (1gb by default) preserve/restore will work the same in Stata16 as it did in Stata15. Use set max_preservemem to change that default to something larger if appropraite.