reshape, greshape, tolong: From which drive does IO system full error occur? Can I change it? Which one is fastest for huge data?

[James Park]

1. I was running reshape on a 30GB data with 60 million observations and 150 variables (6 stub variables). I use Windows 11. My laptop has 68GB RAM.

Then after long hours it says "I/O error writing .dta file Usually such I/O errors are caused by the disk or file system being full."

But I am using data that's stored in D drive, and my D drive has 1.8TB remaining.

So will Stata use some space in C drive (or whichever drive that contains Stata exe file) regardless of the drive that stores the data I am using?

But I can't just suddenly make free space in C. Can I make Stata use D drive instead for performing reshape, greshape, tolong?


2. This post by the creator of tolong compares reshape, greshape, tolong for a small data.

For a 30GB data with 60 million observations and 150 variables (6 stub variables), which command is the fastest?

[Nick Cox]

di c(tmpdir)

tells you where Stata will put temporary files. My guess is that Stata will use somewhere on the drive on which it is installed.

https://www.stata.com/support/faqs/d...ment-variable/ tells you about work-arounds.

[Clyde Schechter]

For a 30GB data with 60 million observations and 150 variables (6 stub variables), which command is the fastest?

-tolong- and -greshape- are consistently faster than -reshape long- when applied to data sets of this size. As between -tolong- and -greshape- I have a sense that -greshape- is the faster of the two, but I have not run any formal comparison testing. So that's just a somewhat vague impression, and mostly based on data sets that are large, but not as large as yours.

[Rafal Raciborski]

Late to the party but tolong does not use tempfiles or preserve/restore; all data operations are performed in memory.