Why was the manual memory management (-set mem-) disabled/discontinued? Are Stata's memory problems of Win32 present under Win64 and Linux?

[Joro Kolev]

Good morning,

I am working on an ancient laptop running Win32, with 4gb of physical memory.

While performing a computational experiment, I hit into an old and very absurd problem:

"op. sys. refuses to provide memory
Stata's data-storage memory manager has already allocated 1504m bytes and it just attempted to allocate
another 16m bytes. The operating system said no. Perhaps you are running another memory-consuming task and
the command will work later when the task completes. Perhaps you are on a multiuser system that is
especially busy and the command will work later when activity quiets down. Perhaps a system administrator
has put a limit on what you can allocate; see help memory. Or perhaps that's all the memory your computer
can allocate to Stata.
r(909);
"

The problem is absurd because I am able to use only 1504/4000 = .376 = 37.6% of my physical memory for work, and apparently Win32 is enjoying itself on the remaining 62.4% of my physical memory space.

The nature of the problem is explained here https://www.stata.com/support/faqs/w...ts-and-memory/
if I understand correctly the explanation: a) Stata needs contiguous space to allocate to its working memory b) Win32 writes to arbitrary addresses thereby fragmenting the available memory.

To complete the description of the situation, I was able to manually -set mem- up and including Stata 11, and since Stata 12 I get in return:
"set memory ignored.
Memory no longer needs to be set in modern Statas; memory adjustments are performed on the fly
automatically.
"

My questions are:

1. Is that arrangement where I am not able to reserve the memory space from the outset not dysfunctional? In my view it is dysfunctional for many reasons:
a) the available contiguous space depends on the state of the system, so the time at which you try to reserve the space matters, and generally the later you try to reserve it from restarting the computer, the worse it gets (because there are more processes running at some random memory addresses, fragmenting your available space)
b) I would prefer to know that I am not able to do something at the outset, and not after my program has run for hours/days/weeks.

More constructively, is there any way to overrule this, and reserve my memory space from the outset like in Stata 11 and earlier?

2. Do the same problems exist under Win64 and Linux? Say if I have 4gb of RAM, what percentage in your experience you can use under Win64 and Linux for work, and what percentage is reserved for the operating system to enjoy itself?

[Alan Riley (StataCorp)]

Although your computer has 4 GB of RAM installed, your 32-bit computer may not be able to use all of it. Here is one article explaining why that might be (and probably is) the case:

https://www.pcworld.com/article/2013...gb-of-ram.html

Stata no longer has the set memory command because it is truly no longer needed. As of the time we introduced Stata's automatic memory management for the data area, Stata stopped allocating memory for datasets as one large contiguous chunk and instead allocates it in segments. You can see this in the error message you received. You didn't receive an error that the operating system refused to provide a large, contiguous, chunk of memory for the entire dataset. You received an error that the operating system refused to give Stata just 16 more megabytes.

That's on the operating system, not Stata. Stata would love to use that 16 more megabytes, but Windows for whatever reason is refusing to let Stata have it.

There are some memory settings you could try tweaking, but none of them are going to solve the root cause of the problem which is that 32-bit Windows is pretty bad at memory management. See [D] memory in the Data Management Reference Manual for details on settings. You could try set min_memory up front, which in a sense is somewhat like the old set memory command in that it will force Stata to try to allocate a certain amount up front. Stata will still try to do it in segments, however (but you can control the size of each segment with set segmentsize), and your 32-bit Windows OS will likely still limit you to something way less than the actual amount of RAM you have installed.

If you were using a 64-bit operating system, you would have no problem getting closer to the actual amount of RAM on your computer.

[Joro Kolev]

Thank you Alan Riley (StataCorp) for illuminating how memory management in modern Stata works, I am pretty sure that this is the first time I hear/read that modern Stata is able to allocate memory in segments. I agree that with a hindsight this should be clear from the error message...

Thank you for the hints on -set min_memory-, I will read the manual and try it. If this really works, this would address my concern that with this allocation of memory on the fly, it happens to me all the time that a program runs for very long time, and then terminates due to memory shortage. With -set min_memory- I would know at the outset that my attempts would later turn futile.