set maxvar or set minmemory?

[Esha Munoz]

Hi everyone,

I'm a new Stata user, so my pre-emptive apologies for any basic questions.

I've just downloaded Stata MP, and am trying to increase the memory available for its use in order to pull together quite a large panel dataset with a veyr large number of observations over 14 waves.

I'm constantly getting this message when I try to set min memory:

set min_memory 240g
op. sys. refuses to provide memory
Stata's data-storage memory manager has already allocated 28g bytes and it just
attempted to allocate another 2g bytes. The operating system said no. Perhaps you are
running another memory-consuming task and the command will work later when the task
completes. Perhaps you are on a multiuser system that is especially busy and the
command will work later when activity quiets down. Perhaps a system administrator has
put a limit on what you can allocate; see help memory. Or perhaps that's all the memory
your computer can allocate to Stata.

Despite this, I have still tried to pull together the panel, and have received this message as Stata works through the variables:

no room to add more variables
Up to 25,000 variables are currently allowed, although you could reset the maximum using
set maxvar; see help memory.

I have changed the location of my Stata program to the D-drive and temp folder, but I'm not having much luck.

I have a Windows operating system. Would anybody please have any advice on what I could possibly do?

Many thanks,

Esha

[William Lisowski]

Welcome to Statalist, Esha.

I have a number of comments on your post. But let me start by apologizing in advance if I've misunderstood anything in your post. I'm making several assumptions "reading between the lines" of what you wrote, and I may be completely incorrect in my thoughts.

I have changed the location of my Stata program to the D-drive and temp folder, but I'm not having much luck.

It appears from this that you believe that Stata's messages and settings about memory refer to the amount of disk storage available or in use. This is not the case: it refers to the amount of computer memory (real and virtual) that it has available for use. Moving your application and files to a different location on disk will have no effect on availability of computer memory.

With that said you apparently have gotten to the point that you have about 25000 variables created and it has hit the limit currently set for the number of variables. You could increase the number of variables using set maxvar as Stata's error message suggests.

You should certainly follow Stata's advice and read the output of help memory to better understand your situation and your options for dealing with it.

But now, let me turn from Stata to what may be the source of your problems, dealing with panel data over multiple waves where the variable names are different in each wave. I work in that area, and I am am guessing you might be going about things the hard way.

First, if you haven't already done so, you should review the Stata Longitudinal-Data/Panel-Data Reference Manual PDF included with your Stata installation, particularly the Introduction to xt commands section, so that you have an idea of how your panel data should be set up for analysis in Stata. You will see in brief that for each individual you goal will be to have separate observations for each wave of data. You will not want to have one observation with 14 values of age, 14 values of income, 14 values of marital status, ... . This is, to Stata, a "wide" layout of your data. The experienced users here generally agree that, with few exceptions, Stata makes it much more straightforward to accomplish complex analyses using a "long" layout of your data rather than a "wide" layout of the same data. What I described as your goal is a "long" layout, and that is the layout that the commands for dealing with panel data expect.

Second, I start by trying to understand what variables I will need for my analysis, and as I read each wave I keep only those variables that I will need. Or, when I use data from a survey that lets me customize my selection of data, I download only the variables I will need from each wave.

Third, as I process each wave, I rename the variables so that the variable for a given concept has the same name in each wave. For example, one survey I use has aage as the variable name for age in the first wave; bage in the second wave, and so on, through oage in the 15th wave. I change these all to age as I process each wave, and in processing the wave I create a variable indicating the wave from which the observation is taken.

Fourth, I append the separate datasets I've created for each wave together to create the file I need for analysis. (I'm thinking you may be trying to merge your waves, which is why you are getting so many variables.)

That's a brief overview of the effective path for analysis of panel data in Stata.

Finally, are you new to Stata? If not, you can skip the rest of this post.

Otherwise, I'm sympathetic to you as a new user of Stata - it's a lot to absorb. And even worse if perhaps you are under pressure to produce some output quickly. Nevertheless, I'd like to encourage you to take a step back from your immediate tasks.

When I began using Stata in a serious way, I started, as have others here, by reading my way through the Getting Started with Stata manual relevant to my setup. Chapter 18 then gives suggested further reading, much of which is in the Stata User's Guide, and I worked my way through much of that reading as well. There are a lot of examples to copy and paste into Stata's do-file editor to run yourself, and better yet, to experiment with changing the options to see how the results change.

All of these manuals are included as PDFs in the Stata installation (since version 11) and are accessible from within Stata - for example, through the PDF Documentation section of Stata's Help menu. The objective in doing the reading was not so much to master Stata as to be sure I'd become familiar with a wide variety of important basic techniques, so that when the time came that I needed them, I might recall their existence, if not the full syntax, and know how to find out more about them in the help files and PDF manuals.

Stata supplies exceptionally good documentation that amply repays the time spent studying it - there's just a lot of it. The path I followed surfaces the things you need to know to get started in a hurry and to work effectively.

Good luck! And let us know if you continue to have problems.

[Alan Riley (StataCorp)]

Hi everyone,

I'm a new Stata user, so my pre-emptive apologies for any basic questions.

I've just downloaded Stata MP, and am trying to increase the memory available for its use in order to pull together quite a large panel dataset with a veyr large number of observations over 14 waves.

I'm constantly getting this message when I try to set min memory:

set min_memory 240g
op. sys. refuses to provide memory
...

Despite this, I have still tried to pull together the panel, and have received this message as Stata works through the variables:

no room to add more variables
Up to 25,000 variables are currently allowed, although you could reset the maximum using
set maxvar; see help memory.

William gave you a lot of useful advice. I would like to point out two things.

1) If you receive errors like the second one you received, simply follow the advice and increase set maxvar to what is truly needed for your dataset.

2) It is very rare for a user to need to set min_memory, and in your case, it isn't helpful at all. The only time a user should set min_memory is when they know for sure their dataset(s) always require that minimum amount, they have enough memory to accomodate that, and they want Stata to reserve that memory from the operating system at the expense of other applications or other users that might be running on the same system.

In your case, you do not need to adjust any memory settings. Stata will automatically determine how much memory is needed for your dataset and allocate as necessary. You'll only get an error about memory if your dataset is larger than your computer can handle.

[Esha Munoz]

William and Alan, a belated sincere thanks to you both - i am indeed new to stata. I ended up creating a long file, selecting only the variables i needed and then balancing it by eliminating individuals with missing observations in an of the waves. i appreciate the time taken to respond to my questions.