Can I improve estimation speed of gsem command using Stata/MP?

[David Bayliss]

Hi,

I have question regarding optimizing the multiple processor features in Stata/MP (13.1). I have access to a high speed shared computing facility at work and can connect to a node which has 64 GB RAM and 12 CPU cores. Using Stata/MP I have run a series of generalised SEM using gsem. I found the models ran at the same speed on my laptop (8 GB single core laptop running Stata/IC 14) as they did on the multi-core system. I continue to use the multi-core system because it is convenient, however my current model has been running for 8 days so far so I am exploring alternatives. My question is, am I doing something wrong which is leading to the models running at the same speed on the multi-core and single core machines?

From what I have read, the multi-core features work automatically - i.e. I don't need to ask Stata to use more cores. I understand Stata/MP can only run as fast as the mathematical processes will allow and therefore doesn't speed up all opreations. I don't however have the statistical background to understand whether my model should be running faster than the 'lower bound (no improvement)' as it says on the Stata website.

The data is unbalanced panel data held in 'long' format with 78,000 observations of 13,000 people. The first two lines of code estimate a continuous outcome variable, while the last four lines estimate binary outcomes. The code is as follows:

gsem(m_y1 <- x0 x1 x2 i.x3 1.x4 1.x4#c.x1 1.x4#i.x3 i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U1[pid]) ///
    (t_y1 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U1[pid]) ///
    (2.m_y2 <- 1.i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 x12 x13 x14 1.x15 x16 x17 1.x18 x19 x20 1.x21 x22 x23 1.x24 x25 x26 1.x27 x28 x29 U2[pid]) ///
    (3.m_y2 <- 1.i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 x12 x13 x14 1.x15 x16 x17 1.x18 x19 x20 1.x21 x22 x23 1.x24 x25 x26 1.x27 x28 x29 U3[pid]) ///
    (2.t_y2 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U2[pid]) ///
    (3.t_y2 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U3[pid]) if sex==1, nocons

Thank you,
David Bayliss

[wbuchanan]

Have you tried fitting a more parsimonious model first to see if you still have the same issues? What information is being printed to the console (e.g., are you getting messages about the LL being non-concave, etc...)? Have you tried fitting the same model using the -difficult- option for maximization? Sometimes allowing Stata to change the ML algorithm can help with performance (at least in my experience).

[David Bayliss]

Hi wbuchanan,

The current model which is taking several days to run follows a series of simpler models, each of which required an increasing length of time to run. The vast increase in time required to run the models comes when the random effects are jointly specified across multiple equations with a freely estimated covariance structure. This specification is central to the model for interpretation. The models all work successfully in that they produce the desired estimates. In the more complex models this occurs after around 20 interations of fitting the full-model, with "(non concave)" LL in most of the first 16 interations, but none after that.

I will explore the -difficult- option.

Thanks for your help.

[wbuchanan]

David Bayliss The "(non concave)" message is telling you that the model is having difficulty converging. There are also options for specifying how starting values are estimated, but it may be a good idea to try stopping the model preemptively and using the estimates as starting values in subsequent iterations. Are you using any system monitoring tools while fitting the model (e.g. top/htop, etc...)? If you start hitting RAM limitations swapping memory can kill the runtime and given the complexity of the model it seems like it could potentially be an issue. Richard Williams might have some helpful ideas since he also does a lot of SEM work in Stata.

[Richard Williams]

I haven't found that MP with sem is all that much faster (indeed, maybe not any faster) than SE. Bill Gould confirmed as much in this thread:

http://www.statalist.org/forums/foru...se-or-stata-mp

I too have tried running sem on some superpowered UNIX machines we have. To my surprise, I found they weren't any faster than my 5 year old desktop. I was told that the unix machines were great with monster data sets (which is indeed the case) but their clock speed probably wasn't that much better than what i had with my own machine.

I decided to splurge on MP because I was starting to do a lot of sem work. I probably would not do so again, especially now that I know how to use these UNIX machines that can do heavy duty work. But for things other than sem MP can have major advantages.

As for other ideas that might help, see this recent thread. I have found that some of the simple tweaks discussed there can sometimes work miracles. Unfortunately, I don't know which tweak is going to work in which situation.

http://www.statalist.org/forums/foru...extremely-slow

I love Stata and I love sem, at least for the smaller models that sem can zip through. But if you have monster models and you really don't want to wait 8 days or more, I would consider getting mPlus. I don't have mPlus myself, but people who have tried both say that mPlus can do in minutes what sem might take several hours for. I've also heard that SAS is quite slow compared to mPlus, so it isn't just a Stata issue. Maybe Stata can buy mPlus and steal their algorithyms! Or maybe use mPlus to get spectacular starting values and then use Stata to take advantage of all its great post-estimation commands.

[Richard Williams]

Incidentally, this should be in the general forum, not the sandbox forum. I never would have noticed this thread if William hadn't tagged me causing me to get an email message about it.

[David Bayliss]

Thank you both for suggestions and links. I'm working through the suggestions. If I manage to increase the speed, I will post the details of how.

[wbuchanan]

David Bayliss the only other thing I could think of doing is to start tweaking individual maximization/starting value options. If your code is stored in a file like:

// mygsem.do

gsem(m_y1 <- x0 x1 x2 i.x3 1.x4 1.x4#c.x1 1.x4#i.x3 i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U1[pid]) /// (t_y1 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U1[pid]) /// (2.m_y2 <- 1.i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 x12 x13 x14 1.x15 x16 x17 1.x18 x19 x20 1.x21 x22 x23 1.x24 x25 x26 1.x27 x28 x29 U2[pid]) /// (3.m_y2 <- 1.i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 x12 x13 x14 1.x15 x16 x17 1.x18 x19 x20 1.x21 x22 x23 1.x24 x25 x26 1.x27 x28 x29 U3[pid]) /// (2.t_y2 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U2[pid]) /// (3.t_y2 <- i.x5 i.x6 i.x7 1.x8 i.x9 1.x10 i.x11 U3[pid]) if sex==1, nocons $maxopts $startopts

// End of do file

You can iterate over this and get a better idea of the timing like:

profiler on

// Define maximization options to test
glo maxopts iter(5) difficult

// Define starting values options to test
glo startopts startvalues(zero)

// Fit the model with the maximization/starting value options specified above
do mygsem

// Report out on the timing of the Stata processes
profiler report

// Clear the profiler's memory
profiler clear

// Set new maximization options
glo maxopts iter(5)

// Set new starting value options
glo startopts 

// Refit the model again
do mygsem

// View the timing
profiler report

It may be a bit of a verbose process, but if you know specific combinations that you think might help you could write a loop to write a do file for you with the different permutations of those optional arguments and then it wouldn't be too bad. I'm also assuming that the iterations are taking similar amounts of time, which likely isn't completely true but it might give you some crude measure of efficiency (with regards to computational time).