What hardwares to upgrade? One regression takes more than 10 minutes.

[James Park]

I have several tens of GBs of datasets. When I include all observations in my regression, just one regression takes more than 10 minutes. I already have 128GB-RAM computer. But I wanted to make it even better.

But if I open Task Manager when my computer is running the regression, still up to 25% of Memory is being used. Does this mean upgrading my computer to 256GB-RAM will be useless? Why does it look like my computer is not using the full potential of memory that it already has?

[Leonardo Guizzetti]

It's hard to give specific answers for your particular data and regression. It seems like you have more than enough RAM. A version of StataMP with complemented by more cores and an overall faster processor would help, but I think only marginally. A solid state hard drive could also boost performance because it may take a lot of time to read data from magnetic media.

That said, ten minutes is not so long to wait given the size of your dataset. I've run regressions of survey data with ~300M observations that have taken hours.

[Sven-Kristjan Bormann]

You should tell us the number of observations, how large your dataset is and which command you are running.
The amount of RAM should not matter for how long the regression takes to estimate. The RAM just matters for holding the dataset and all temporary objects(variables, matrices, etc.) which are created during the estimation.
Depending on the estimation procedure and estimation problem, you could try to parallelize the estimation to make better use of your CPU. The specifics how to do that depend on whether you use Stata/MP or not and how many cpu cores you have. You could look at the description of the -parallel- user-provided command via

view net describe parallel, from(http://fmwww.bc.edu/RePEc/bocode/p)

in Stata. This could give you an idea how to speed up the estimation process.

[James Park]

Thank you to both!

Here are necessary information.


Stata

Stata MP 10-core


What I am running

command = ivreghdfe with cluster and weights

Number of observations = 10 million


Computer

Processor : Intel Core i9-7900X CPU @ 3.30GHz (10 Cores / 20 Threads)

RAM : 128GB



When the number of observations is just 1 million, it was not a big deal. But with 10 million, it's too slow.

What will be the best solution for this case hardware-wise? I am learning about parallel. It shows me errors but I am trying.

[Leonardo Guizzetti]

I'm afraid there's probably not much more you can do, as your hardware is already quite fast and you're using StataMP/10. The reghdfe commands are already fairly optimized for speed. My guess would be you have a large number of clusters that requires more intensive matrix algebra.

Why is 10 minutes too slow, exactly?

[Sergio Correia]

command = ivreghdfe with cluster and weights
Number of observations = 10 million

Well, ivreghdfe is not as optimized as it could be, but I can't give more advice without knowing more details, such as how are you setting up the fixed effects. EG: If you have FEs id1 and id2, and id1 has 5,000 levels while id2 has 100,000 levels, then the fastest thing to do would be to pre-sort the data by id2 and run ", absorb(id2 id1)".

About the computer, well more RAM/memory does not make Stata run faster unless it is memory constrained, which is not in your case. Further, the benefits of more cores diminish pretty quickly so if you went from Stata MP10 to Stata MP20 (if that even exists) you would only see a marginal improvement. What might matter is a) RAM speed, and b) CPU single-core performance (closely related to speed, but less so nowadays).

[James Park]

Well, ivreghdfe is not as optimized as it could be, but I can't give more advice without knowing more details, such as how are you setting up the fixed effects. EG: If you have FEs id1 and id2, and id1 has 5,000 levels while id2 has 100,000 levels, then the fastest thing to do would be to pre-sort the data by id2 and run ", absorb(id2 id1)".

About the computer, well more RAM/memory does not make Stata run faster unless it is memory constrained, which is not in your case. Further, the benefits of more cores diminish pretty quickly so if you went from Stata MP10 to Stata MP20 (if that even exists) you would only see a marginal improvement. What might matter is a) RAM speed, and b) CPU single-core performance (closely related to speed, but less so nowadays).

Thank you for your answer. Does your tip apply always when there are multiple fixed effects (ID1 and ID2)? And by "level" you mean the maximum categories created by the FE? Then, if ID1 is NAICS 1-digit, then there will be 9 categories, and if ID2 is age of workers, there will be at best 100 categories, so ID2 has more categories than ID1, so I have to pre-sort by ID2 like the following?

sort ID2
ivreghdfe ~~~ ,absorb(ID2 ID1)

And you wrote ID2 first inside absorb. Was that intentional? That is, does the order inside absorb matter for the speed ( absorb(ID2 ID1) vs. absorb(ID1 ID2) )?

Please let me know if I understood you correctly.

[Richard Williams]

I assume you have Stata 16? If not you should upgrade to it.

I suspect many of us would be euphoric if our longest jobs only took 10 or 12 minutes. I know I would be.

When working with monster data sets, I often take a subsample for my preliminary runs, and then switch to the full sample when ready. (Although for many purposes a 10% subsample may be perfectly fine for the final work.)

I also sometimes send monster jobs off to these unix machines that are somewhere off in the distance. The job may or may not run more quickly but my main machine is not getting tied up.

[James Park]

I assume you have Stata 16? If not you should upgrade to it.

I suspect many of us would be euphoric if our longest jobs only took 10 or 12 minutes. I know I would be.

When working with monster data sets, I often take a subsample for my preliminary runs, and then switch to the full sample when ready. (Although for many purposes a 10% subsample may be perfectly fine for the final work.)

I also sometimes send monster jobs off to these unix machines that are somewhere off in the distance. The job may or may not run more quickly but my main machine is not getting tied up.

Oh Is Stata 16 faster than Stata 15? I just googled and are you referring to data frames feature?

[Richard Williams]

https://www.stata.com/features/overv...ry-management/

In Stata/MP, the memory manager can even preserve temporary copies of datasets in memory for speed. And all flavors of Stata support frames, which are multiple datasets in memory.