Stata 16 taking up entire physical memory of a computer

[Scarlet Chen]

I am using Stata 16 MP on a new iMac with macOS Majove (10.14) and noticed in the Activity Monitor (an application that comes with a mac) that Stata is taking up >31gb memory out of a total of 32gb, which is a surprisingly large to me.

To give more context, I was using a .dta of size ~32gb and was using for-loops to run regressions with reghdfe command and plot graphs with binscatter command and I did not use `set memory xxx' to change the default memory setting.

I have not monitored stata memory usage on stata 15 or earlier versions of macOS, so I am not sure whether it's a stata16 issue or a macOS 10.14 issue.

I know that I could change the default memory setting with `set memory xxx', but I'm just wondering whether it could be a bug with stata16 - as it's just out - that there's something wrong with the memory management that it could take up the entire physical memory of a computer.

Many thanks for your help! I really appreciate it!

[Leonardo Guizzetti]

The way Stata works with data is to first load/create the datasets all into memory (or as much as it can). Your dataset is larger than the available memory and so it will use all of it and cache the rest to your hard drive. Unfortunately, working with large datasets means that you need more memory or more patience while it works with the data.

While I have not tried it myself, I think you can tell Stata the max memory size to use, but the old option -set memory- is deprecated and Stata no longer obeys that option from older generations of the software.

[Richard Williams]

So what exactly is the problem with Stata doing this? Is it keeping you from doing other things?

See -help memory-. Perhaps the max_memory or niceness commands will address your concerns.

[Bjarte Aagnes]

It is recommend that your computer have 50% more memory than the size of your largest dataset
see https://www.stata.com/support/faqs/w...-requirements/

You may load a subset of data see -help use- and reduce the amount of memory used by running -compress-

[Richard Williams]

I was going to add that if I routinely worked with 32 GB files I would try to get 64 GB memory (or more). That is what I have. It is massive overkill for most of what I do but it is there when I need it.

But again, I'm not sure what the problem is with Stata taking up most of the memory available. It will run more quickly that way.

[Leonardo Guizzetti]

I was going to add that if I routinely worked with 32 GB files I would try to get 64 GB memory (or more). That is what I have. It is massive overkill for most of what I do but it is there when I need it.

But again, I'm not sure what the problem is with Stata taking up most of the memory available. It will run more quickly that way.

At the limit it will slow down all other processes running on the computer and be relatively unusable while Stata is busy.

[Scarlet Chen]

Thanks so much for all your input! This is super helpful!

The reason why I posted the question is that I'm using an iMac that I bought 2 weeks ago but ran into system freezes: I would leave some program (e.g. python or stata) running before I leave my office and every morning when I come into the office I would find that the computer has frozen - sometimes I couldn't even wake it up from a lock screen; sometimes I can wake it up but most of the programs (e.g. chrome, preview, etc) would have `quit unexpectedly'; sometimes when I click on an application (e.g. safari) it says `the application is not found'; but after I restart the computer everything is ok again.

So I was trying to debug what was the issue. I have had the apple store ran a hardware test and the result says hardwares are ok, so it's likely a OS or software problem. So I started to try to replicate the bug and watch Activity Monitor - the idle CPU is always above 85% so that part seemed ok, but the memory usage is near the limit when I was running stata, so I realized it's likely because the system was out of memory...

And now that I think about it it's obvious that I should not have had stata run a dataset that's close to or above the system's memory in the first place (and should have checked the dataset size beforehand)! So it all makes sense now!

Thanks again and I really appreciate your input!

[Richard Williams]

Edit: I had not seen Scarlet’s follow up post before I wrote this.

To answer the original question, I think Stata’s memory management would be considered a feature, not a bug. It saves you the hassle of managing memory yourself and runs the job as efficiently as possible.

The machine does seem underpowered for the task. Get more memory if possible. If not possible consider the various tips offered so far, like working with a subset of the data or tweaking the settings for max_memory and niceness.

Also, I often do preliminary analysis using only a subset of the data. If one machine is doing a monster analysis I will try to use another. I luckily have the option of submitting huge jobs to Unix machines that are off in the distance somewhere. I sometimes run big jobs overnight. The more things you try to run simultaneously on one machine the longer the monster job is going to take.

The Original poster never did say that they were actually having problems though. So I might not do anything until I did.

[Scarlet Chen]

Thanks so much Richard! This is very helpful!

Sorry for bringing up a slightly unrelated issue: I just noticed that even after stata stopped running (my code was killed pre-maturely due to a bug), the memory usage of stata in Activity Monitor is still >31gb. My prior was that when stata is actively running a piece of code it may take up a large amount of memory depending on how large the dataset that it's working on is, but when a code has stopped, stata should have stopped running and in principle the memory usage should drop to a much lower value? Or is my understanding of this wrong?

Sorry for not bringing this up earlier - I just noticed this and found it puzzling and thought it may be worth posting for feedback. Again, I really appreciate all your input and help!

[Richard Williams]

Did Stata actually exit or did it just stop running the job and give an error? If Stata is active and the data set is still in memory then I assume it will take up space whether anything is running or not.

i also assume that if you give the clear command all that memory will become available, but you can try it and see.

if Stata has crashed and exited and is still eating up 30 GB memory, that is annoying. Maybe you should start Stata and quickly exit. Or just reboot your machine. With Windows my theory is that 90% of all weird problems can be solved by turning the machine off and then on again.

By the way, welcome to Statalist.