Large Dataset Memory Issues

[Claire Davies]

Hello, I am currently attempting to do analysis of census data so have a very large dataset. It has 183,467,917 observations, size 5.1 GB, Memory 6.5 GB and the file size is 8,241,738 KG. Obviously this is extremely slow to run commands. I have dropped variables and compressed, is there anything else to do? Would using an a physical SDD drive help? Any ideas would be great thank you.

[Sergiy Radyakin]

Hello Claire,

6-8GB data doesn't sound 'large' these days and modern Stata versions can handle that effortlessly.

You need to make sure that you have plenty of room in RAM for temporary variables, additional frames, etc and leave ample space for parallel programs and OS itself to avoid swapping to your drive.

You've asked:

> Would using an a physical SDD drive help?

Is that even a question? An SSD will greatly enhance the performance of the IO-operations, though it is not clear how this will impact your overall work, as not all commands do IO.

- get a computer with decent amount of CPUs (even if you have a single CPU Stata license, offloading other processes to other cores should help use it more efficiently).
- get an MP-version of Stata. Usually the more the better, though not always linear increase in performance.
- get decent amount of RAM. 32GB may be acceptable (not optimal) for working with this size of data.
- switch off swapping to drive at the OS level
- consider RAID for boosting your performance (RAID 0).
- check your code. minimize preserve/restore or commands that internally rely on preserve/restore.

Overall, consider your strategy. Rather than running commands on the full dataset, take a 1% sample and program your analysis fully as a do-file(s). Once you complete programming, leave it overnight for tabulations running on the full dataset. If you don't get your census tables by morning, you probably need to describe some more / provide details of what you are doing.

Best, Sergiy

[Daniel Feenberg]

I have some advice for large datasets at http://www.nber.org/stata/efficient but it depends on what you are doing. Some Stata commands will create a temporary working dataset with every variable upgraded to double - if that causes your job to page it will be very slow. In general 5GB is not so large that you should expect problems

[Leonardo Guizzetti]

Here is some other practical advice to consider while taking a wild guess at what your are actually trying to do. I reasonably guess you are using an average retail computer, and are attempting to do fairly standard data manipulation or analysis techniques (e.g., regression modeling). It is quite likely that your biggest bottleneck with a dataset of that size is your RAM, which is probably 8 or 16 GB. You will need at least twice size of your dataset in free RAM to comfortably let Stata work with this data. Second, the advice about dropping unnecessary variables and using compress use the smallest fidelity storage types is also quite helpful. Hard disk type (SSD vs magnetic HDD vs RAID arrangements) will only be relevant for disk IO operations for reading and saving datasets to disk. This could be a slow operation but may be manageable given you probably may not have any IO steps. Lastly, when I have had to work with with large datasets of your size (or bigger), I always take a tiny representative sample for developing code, maybe 5-10% (more or less, as needed). This lets me have a dataset that is reasonably quick to load and use so I can write and test code in reasonable amounts of time, and then use the full dataset only once I was reasonably sure of the code. I recommend doing this sampling process once and saving it so you don't need to load the entire dataset every time.

[Daniel Feenberg]

Sorry, that URL is acutally http://www.nber.org/stata/efficient but you should provide more information about what procedures are slowing you down.

As I and others have stated, if you run out of memory things will be very slow for all procedures. Stata does not have a way to check for memory usage within a command, but the WIndows task monitor will let you see if that is happening. My webapge has some suggestions when the compress command isn't sufficient.

[Leonardo Guizzetti]

One more thought, and it should be completely obvious, analysis procedures on big data necessarily takes longer than smaller datasets. It may not be unreasonable for it to take minutes, hours or days for models to to be fit depending on the complexity.

[Mike Lacy]

There is a lot of good advice here. I'd also add that some analyses (e.g. ones involving pairing of individuals in an explicit or implicit way) can require run time that increases as the square of the sample size, or worse. In situations where I suspect that, I might try a sample of say 5,000 observations, and then 10,000, and see how the run time increases. At least that would give you an idea of what to expect with the full sample.

In the absolute worst case, if there's no reasonable way to analyze the whole data set at once, you could compare the parameter estimates for several different different samples of only (say) 100,000 observations each, and hope that they all showed similar results. Some years ago, when computers were smaller and slower, some colleagues and I published an article in a good journal with an approach like that; I think we reported the mean of the parameter estimates for each explanatory variable.

[Bruce Weaver]

In the absolute worst case, if there's no reasonable way to analyze the whole data set at once, you could compare the parameter estimates for several different different samples of only (say) 100,000 observations each, and hope that they all showed similar results. Some years ago, when computers were smaller and slower, some colleagues and I published an article in a good journal with an approach like that; I think we reported the mean of the parameter estimates for each explanatory variable.

Alternatively, you could use standard meta-analysis to compute pooled estimates of the coefficients with confidence intervals.