I'm trying to estimate the competing risks hazard model of Fine and Gray (1999) using the stcrreg command in Stata/MP 15.0, running on a ~40 core Linux server with ~500GB RAM. My data are fairly large, consisting of a panel of delinquent mortgages, with one observation per loan per month. The outcome of interest is foreclosure, with competing risks of pre-payment (selling the house or refinancing) and curing (coming current on the loan). We do need the panel structure as there are several important time-varying covariates.
The problem is that the stcrreg command seems to run very, very slowly despite having plenty of RAM and computational power available. Running on a 0.1% sample of our data, with about 15,000 observations on ~2000 loans (about 230 failures), it takes more than 3 hours for the model to converge. With a 1% sample, the model ran for at least 24 hours before we killed it. By comparison, ignoring the competing events and estimating the same specification with stcox takes less than 5 minutes even with the 1% sample.
Anyone know why this happens, and what the best way is to deal with it? Is it just that stcrreg was never parallelized, or otherwise poorly optimized?
I've come across the stcrprep package by Peter Lambert on SSC, which by its description sounds like it might speed things up, but I'm puzzled at the overall slowness in the built-in package.
Thanks,
-Ryan Sandler
The problem is that the stcrreg command seems to run very, very slowly despite having plenty of RAM and computational power available. Running on a 0.1% sample of our data, with about 15,000 observations on ~2000 loans (about 230 failures), it takes more than 3 hours for the model to converge. With a 1% sample, the model ran for at least 24 hours before we killed it. By comparison, ignoring the competing events and estimating the same specification with stcox takes less than 5 minutes even with the 1% sample.
Anyone know why this happens, and what the best way is to deal with it? Is it just that stcrreg was never parallelized, or otherwise poorly optimized?
I've come across the stcrprep package by Peter Lambert on SSC, which by its description sounds like it might speed things up, but I'm puzzled at the overall slowness in the built-in package.
Thanks,
-Ryan Sandler
Comment