Can I replicate Stata 17 speed improvements using Stata 16?

[James Park]

https://www.stata.com/new-in-stata/f...-improvements/

Stata says Stata 17 made many improvements for speed.

They say they improved the speed of sort and collapse commands. But is it possible to produce the Stata17-level speed by using Stata16? Like, someone made "tolong" command to make "reshape long" faster.

Stata also says "we also attained speed improvements for estimation."

Does that include regression commands often used by economists like regress, ivreg, reghdfe, ivreghdfe ?

[Nick Cox]

Whatever the company did by changing ado code could in principle be done by a user-programmer.

Whatever the company did by changing internal C code is impossible for a user-programmer to mimic exactly.

reghdfe, ivreghdfe are user-written (community-contributed) commands (from SSC) that may run faster in 17 insofar as they use Stata commands internally that have been speeded up. My impression is that such programs typically use Mata directly rather then e.g. regress.

StataCorp code sometimes runs slower than apparently competitive community-contributed code for reasons like

1. There are more checks on syntax and data. Checks on syntax are often not time-consuming but checks on data may be.

2. Testing has gone much further than users typically do with their programs. A user-programmer (me too) is usually happy at getting results from their own data and doesn't push hard on whether the code works well, or indeed at all, in extreme or pathological cases that might arise. User-programmers are not much embarrassed if someone else's data raises a problem they had not encountered themselves or failed to imagine.

More authoritative answers may be hoped for from the writers or users of such commands.

[James Park]

reghdfe, ivreghdfe are user-written (community-contributed) commands (from SSC) that may run faster in 17 insofar as they use Stata commands internally that have been speeded up. My impression is that such programs typically use Mata directly rather then e.g. regress.

The explanation for the speed of Stata 17 says

"For Stata 17, we also attained speed improvements for estimation. The Linear Algebra Package (LAPACK) underlying many of Mata's functions and operators is now powered by Intel Math Kernel Library (MKL). How much faster is the new MKL? Multiplying a 5,000-by-5,000 real matrix in Stata/SE with a real matrix of the same dimension takes about 13 seconds using MKL in Stata 17 compared with 70 seconds in Stata 16."

[James Park]

Oh I didn't complete my question.

The explanation for the speed of Stata 17 says

"For Stata 17, we also attained speed improvements for estimation. The Linear Algebra Package (LAPACK) underlying many of Mata's functions and operators is now powered by Intel Math Kernel Library (MKL). How much faster is the new MKL? Multiplying a 5,000-by-5,000 real matrix in Stata/SE with a real matrix of the same dimension takes about 13 seconds using MKL in Stata 17 compared with 70 seconds in Stata 16."

My question is whether this MKL will make commands like regress, ivreg, ivreghdfe faster.