op. sys. refuses to provide memory

[salma ktat]

Hi,
I have a 4Go RAM computer with 64bits operating system. I want to read a huge daily dta. file but stata said no room to add more observations. so I tried to set the memory and increase it but it keep saying op. sys. refuses to provide memory. My current memory is 1000.00 MB allocated to data.
So, how can I increase the memory allocation knowing that stata just refuses to do it!
Thanks.

[ben earnhart]

What version of Stata? And is it a 64-bit or 32-bit installation? Also, a reboot is a cheap, simple solution that solves a remarkable amount of these issues.

[salma ktat]

The version is stata 11.2 and it's a 64-bit installation. And I reboot it but the problem persist!

[ben earnhart]

try the reboot? Should be able to grab more memory.

[salma ktat]

I did but it's not working!

[Sarah Edgington]

What are the results of typing memory query?

[Nick Cox]

Assuming that your reboot did not involve starting up Stata this problem may have no solution beyond adding (much?) more memory or using a bigger machine.

But tell us more about the data file. Is it a Stata .dta file created somewhere else? Is it something else? How big is it? Do you need all the data in it or can you read in just some variables?

[salma ktat]

It's a dta file downloaded from my database. It's a 6,48 Go file which contains the daily stocks prices I'll be using to calculate the daily returns and then the annualised returns. So, yes I guess I need all the variables in there.
And this is the result when I type query memory

current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.909M
set memory 1000M max. data space 1,000.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
1,003.163M

[Nick Cox]

I take it "Go" means gigabyte or GB. If so, your combination doesn't sound workable. Consider that you not only want to read it in, you want to create extra variables, and so make a very big dataset even bigger.

Others may have better news.

[Sergiy Radyakin]

Hi,
I have a 4Go RAM computer with 64bits operating system. I want to read a huge daily dta. file but stata said no room to add more observations. so I tried to set the memory and increase it but it keep saying op. sys. refuses to provide memory. My current memory is 1000.00 MB allocated to data.
So, how can I increase the memory allocation knowing that stata just refuses to do it!
Thanks.

Salma,

1. Read the FAQ - link at the top of the page.
2. Follow the FAQ to tell the statalist members reading your post what Stata you are using, what commands did you type and what Stata responded.
3. Read what Stata has responded : Operating System refuses to provide memory. Not Stata. Hence you need to configure your system properly. Proceed to the corresponding forum
4. If your super-secret operating system, which you don't name is Windows check the memory limits: even 64-bit installations may not use all the memory available. For other operating systems that is likely also true.
5. If you are using university computers - consult the system administrator, he is getting paid for this and he is the person who would know if any additional restrictions were imposed on the system.
6. If your operating system is Windows - call Redmond, you've paid money, you are entitled to support. I did this once. Never ever again. I'd rather go against the Nemean Lion one-on-one.

My diagnosis is that your system is configured not to use swap file or use a limited one, and you are trying to allocate more than available. This agrees with all the symptoms. Second best diagnosis is that you have a 32-bit Stata despite you are writing it is 64-bit installation. This agrees with inability to allocate more than 1GB.

Set memory to the largest amount that Stata can, tell us what is that value.

Finally, we have no idea what you have typed. Perhaps you are trying to allocate a few godzillions of petabytes. Make sure you specify -set mem- to exactly what you need.

Best, Sergiy Radyakin

@Nick: GO=GB: http://www.acronymfinder.com/Giga-Oc...-%28GO%29.html

[ben earnhart]

I have a variety of ideas, depending on whether the data is long or wide. And do you need to keep the daily returns, or are they merely a way to get to annualized returns? If it's 6.5 GB to begin with, and you need to double it (daily returns), you have a problem. But if you just need them as a way to get to annualized returns, there are options.

[salma ktat]

My operating system is Windows7 and it is 64-bit installation. The max memory I'm able to set is 1000MB. And the file I'm trying to use is a 6,48 GB file which contains the daily stocks prices I'll be using to calculate the daily returns and then the annualised returns. And I mainly using it to get the annualised returns.
The format of my file is as follow: id date price factor dividend etc.

[Sergiy Radyakin]

In all probability you have a 32-bit Stata installation running on a 64-bit Windows installation.

Note also that there are 6 different editions of Windows 7 and a non-trivial number of versions.

Open Stata by clicking it's icon in the Start menu, put the cursor in the little window titled "Command". Type one word: about and press enter. Stata will write some letters in the window titled "output". Use mouse to highlight the response of Stata and copy it to the clipboard, then paste here.
Show the commands and output, how you determined the value 1000MB. The value looks suspicious. Have you tried 1001MB?

If you can't open the data, there is no point in discussing whether it contains daily or weekly data, etc.

There is a possibility to work with partial data (say you load only first year data, or 1st stock data, then the next one, etc), see help use or here.

Best, Sergiy.

[ben earnhart]

I wrote some code a while back to do -infile- on a monster text file containing strings, that not only made it possible, it ran in a fraction of the time they eventually accomplished by bumping the server up to 64 GB. If you have a list of codes, it might look something like:

foreach code in  ABCDA ABCDB ABCDC ABCDD {
use myfile if ID== `code' 
sort date
  [do stuff to get your annualized returns, I dunno if you compound]
keep [annual returns?   important variables?   regardless cut things down, cases and variables]
save temp_`code', replace
}

then run another loop to append the important parts.

I do think that Sergiy is correct, but if you can't re-install Stata, a work-around of the type I describe *might* get the job done.

[salma ktat]

In all probability you have a 32-bit Stata installation running on a 64-bit Windows installation.

Note also that there are 6 different editions of Windows 7 and a non-trivial number of versions.

Open Stata by clicking it's icon in the Start menu, put the cursor in the little window titled "Command". Type one word: about and press enter. Stata will write some letters in the window titled "output". Use mouse to highlight the response of Stata and copy it to the clipboard, then paste here.
Show the commands and output, how you determined the value 1000MB. The value looks suspicious. Have you tried 1001MB?

If you can't open the data, there is no point in discussing whether it contains daily or weekly data, etc.

There is a possibility to work with partial data (say you load only first year data, or 1st stock data, then the next one, etc), see help use or here.

Best, Sergiy.

Hi Sergiy,
You're right. It's a 32-bit stata installation running on a 64-bit windows installation. And yes I tried 1001MB and the system refuses to give memory. This is the output when I wrote about in the command window of stata.

Stata/SE 11.2 for Windows (32-bit)
Born 16 Feb 2012
Copyright 1985-2009 StataCorp LP

Total physical memory: 4140912 KB
Available physical memory: 2230100 KB

Single-user Stata license expires 31 Dec 9999:
Serial number: 71606281563
Licensed to: STATAForAll
STATA

So any suggestions please!