"I/O error writing .dta file" error

[Paul Mueller]

Hello,

I'm using Stata SE 13.1 on Windows 7 and my PC has 8 GB memory and about 374 GB of free space on the hard drive.

I'm working with a data file that has more than 41,000,000 lines. Some commands just won't work, for example "sts".

I keep getting the following error, even when I set the limit to the first 10,000 lines:

I/O error writing .dta file
Usually such I/O errors are caused by the disk or file system being full.
r(693);

However, the command works if I use a small file.

I really need to get this working. The Stata help file doesn't provide any help about this error. Any ideas?

Thanks in advance!

[William Lisowski]

We really need more to go on here. Please read the Statalist FAQ linked to at the top of this page, especially section 12, for advice on the sort of information that helps the reader better understand the problem you're having. For example, you don't tell us what command yielded the error message, and it's apparently not save since you "set the limit to the first 10,000 lines" which isn't an obvious option for save. Please copy and paste sufficient commands and output from the log for us to see what you are attempting. Also, with your 41,000,000 observations in memory, run the describe, short command at the point just before where you would run the command that fails, and post those results to give us some idea of the actual size of your dataset at the point of the problem. If you don't have a large number of variables, omit the short option. Also, run the about command and include those results in your post so we learn what version of Stata you are using.

And finally, with all that copying and pasting, please follow the instructions in the FAQ on using the forum software to place the information pasted between CODE delimiters

like this text
and this text

to maximize its readability.

[Paul Mueller]

Hi William,

thanks for your answer. As mentioned before, I get the error message (among others) from the "sts" program which contains a number of tests and graphics for survival data. I first tried "sts graph" but later I recognized that any option of "sts" leads to the same error. I tried

sts graph
sts graph if dummy==1
sts graph in 1/10000

and in order to identify the problem I also tried

sts

Here's the information about my Stata installation:

. about

Stata/SE 13.1 for Windows (64-bit x86-64)
Revision 30 Oct 2013
Copyright 1985-2013 StataCorp LP

Total physical memory:     8232504 KB
Available physical memory: 2988244 KB

And my data file:

. describe, short

Contains data from (...)
  obs:    41,390,318                          (...)
 vars:            37                          25 Jun 2015 10:56
 size: 3,435,396,394                          
Sorted by:  persnr  spell

It's a very typical spell data file that has been stset.

[William Lisowski]

Hi Paul -

Thanks for the additional information, clearly presented. It really does make a difference. Also, you (perhaps unintentionally) suggested other problems when you wrote

As mentioned before, I get the error message (among others) from the "sts" program

Do you mean, with the "among others", to indicate that there are other error messages that occur earlier in your sts processing? If so, the error message you quoted could just be a side-effect of the earlier problem. On the other hand, perhaps you meant that you receive other, routine messages from sts before the error message.

I notice that you are apparently not running the latest updates to Stata 13.1. Try running the update all command, perhaps you are experiencing a problem that was already fixed.

I see from sts.ado that it uses preserve and tempfile commands to save your original data and to create temporary intermediate data files. It's possible that the problem lies in the Windows directory used to hold temporary files. But I'm not a Windows expert, perhaps one of the other readers has experience with temporary filesystem issues on Windows.

I don't think think your use of the if clause is doing as much as you hope to reduce any issues surrounding the quantity of data in trying to localize the problem. I suggest you try actually deleting observations from memory and running your graphs.

keep if dummy==1
sts graph

or

keep in 1/10000
sts graph

Finally, I note from sts.ado that your bare sts command is equivalent to sts graph, so that test apparently doesn't add anything extra. But I absolutely understand trying anything to see if it makes the problem go away.

[Paul Mueller]

Thanks, that helps a lot. Unfortunately, I don't have the necessary Windows permissions to update Stata. But I will try to solve that problem.

I can follow your argument that the problem probably lies in the fact that Stata can't store its temporary files. The computer has two partitions and the actual Windows partition is almost full while the partition where my data are stored has plenty of free disk space. Is there any way to tell Stata to use another directory for temporary files?

[William Lisowski]

Here's a FAQ turned up in the output of the search temporary command. You will see that the temporary directory is indeed in your Windows partition rather than your data partition, so that almost certainly is the source of your problem. The FAQ gives ideas on how to circumvent the problem, or you could ask the support folk to expand your Windows partition.

http://www.stata.com/support/faqs/da...ment-variable/

[Paul Mueller]

Yes, it looks as if the IT support has to take care of this. Thanks a lot!