Speed test - Mata vs. Matlab

[Randy Chugh]

Hello,

As a simple test of Mata vs. Matlab performance, I wrote a Mata function to count from 1 to "n" and an analogous Matlab function. Matlab appears to be much faster than Mata - about 60x faster! For example, when I ask Matlab to count to 1 billion, it takes about 1 second and when I ask Mata to count to 1 billion, it takes about 60 seconds.

Here is the .mata file (saved as "timerfunction.mata"):
mata
void function timerfunction(real scalar n){
real scalar value
value=0
timer_clear()
timer_on(1)
for(i=1;i<=n;i++){
value=value+1
}
timer_off(1)
printf("Mata counted to %5.0g and it took %6.2f seconds.",value,timer_value(1)[1])

}
mata mosave timerfunction()
end

And here is the result of running with n=10^9:
run timerfunction.mata
mata: timerfunction(10^9)
Mata counted to 1.0e+09 and it took 67.90 seconds.


Here is the analogous .m file (saved to "timerfunction.m"):
function timerfunction(n)
value=0;
tic;
for i=1:n
value=value+1;
end
totaltime=toc;
fprintf('Matlab counted to %5.0g in %6.2f seconds.\n',value,totaltime)

end

And here is the result of running with n=10^9:
>> timerfunction(10^9)
Matlab counted to 1e+09 in 1.22 seconds.


Any thoughts on why Mata is performing so poorly compared to Matlab?

Thank you!

[Mike Lacy]

My modest experience with Mata indicates that anything that entails repeated interpretation of a command will be slow, an explicit loop being a typical example. In your case, removing the one statement in your loop gives essentially the same run time. (This feature of Mata frustrates me. Some other modern interpreted languages seem to avoid this issue with loops.)

In defense of Mata, though, it's (as you likely know) much faster when the desired thing is written as a matrix operation. Given that calculations that can be written as matrix operations often take most of the time in a substantial calculation, a comparison of a matrix vs. loop in Mata vs. Matlab would be interesting. I haven't used Matlab, but if you can post the results of a simple example, such as what follows, that would be interesting. On my machine, the matrix versions runs about 12X faster than the loop.

mata:
void function testmatrix(real scalar n) {
   A = runiform(n,1)
   sum(A)
}
void function testloop(real scalar n) { 
   A = runiform(n,1)
   s = 0
   for ( i = 1; i <= n; i++) {
      s = s + A[i,1]
   }
   s
}

end
//
mata: 
timer_clear()
timer_on(1)
testmatrix(1e8)
timer_off(1)
//
timer_on(2)
testloop(1e8)
timer_off(2)
timer()
end

[daniel klein]

I have no formal background in programming, but the task at hand could be rewritten as

void timerfunction2(real scalar n)
{
    real scalar value
    
    timer_clear()
    timer_on(1)
    
    while (--n) ++value
    
    timer_off(1)
    printf("Mata counted to %5.0g and it took %6.2f seconds.",value,timer_value(1)[1])
}

which will speed things up by a factor of 5. Rewriting as

void timerfunction3(real scalar n)
{
    real scalar value
    
    timer_clear()
    timer_on(1)
    
    value = sum(J(n,1,1))
    
    timer_off(1)
    printf("Mata counted to %5.0g and it took %6.2f seconds.",value,timer_value(1)[1])
}

will speed things up by a factor of 20.

So, perhaps Matlab has a better compiler -- at least for this specific task. I'm not sure your test generalizes to anything other than your code, exactly as you've written it.

[Mike Lacy]

The "while (--n) ++value" is something I never would have thought of (thanks daniel klein) , and it illustrates a good point: If you know the right (i.e., "Mata-ish") way to do something, it can make a big difference in speed. The thing that frustrates me here is that a 60+ year old programming construct (an explicit loop) can't be interpreted or compiled to something efficient in Mata. Sometimes these sorts of things arise in Mata because the user (i.e., me) has failed to think in a vectorized way, which is our fault.. But the fact that using -while- and "--" and "++" is faster than a very standard kind of loop construction seems like something about which one can reasonably complain. Perhaps someone(s) here with knowledge about modern compiler/interpreter design can explain.

In any event, I've seen enough things like this in Mata that I'd welcome another article on "Mata Speed Tricks" There's one at
https://scorreia.com/blog/2016/10/06/mata-tips.html, but I'd say that more would be good. I'd particularly appreciate something that has general principles, as opposed to a bag of tricks.

[Joseph Coveney]

. . . when I ask Matlab to count to 1 billion, it takes about 1 second and when I ask Mata to count to 1 billion, it takes about 60 seconds.. . . Any thoughts on why Mata is performing so poorly compared to Matlab?

I think the key—as Mike mentioned and as Daniel implied in his second example—is to think in terms of vectorization of the operation where possible.

For example, here, because it's just counting up to some value, I would use Mata's range operator (I'm not familiar with it, but I wouldn't be surprised if Matlab has something similar) and break the problem into bite-sized pieces (vector lengths) in order to avoid disc caching. Something like the following.

version 19

clear *

mata:
mata set matastrict on
mata set matafavor speed

void function extend(real scalar a, real scalar n) n = (n::n+a)[1+a]

a = 100000
n = 1

timer_clear()
timer_on(1)
do {
    extend(a, n)
} while (n < 1e9)
n = n - mod(n, 1e9)
timer_off(1)

printf("Mata counted to %5.0g and it took %6.2f seconds.",
    n, timer_value(1)[1])

end

exit

On my antique laptop, I get 1.01 seconds on the first pass and 0.96 seconds on a repeat, and so we're in the Matlab range already with just this off-the-cuff thing. You could probably tweak it further by playing with the vector length setting (the function's argument a) or switching to a row vector with

void function extend(real scalar a, real scalar n) n = (n..:n+a)[1+a]

I can't remember whether Mata is column major or row major (I think the latter in that it's C-like), but it might not make much difference if behind the scenes Mata optimizes the vector formation and does the vector operations to its own design's best advantage.

Regardless, I'm a little leery of relative speed in benchmark tests on arbitrary problems as the primary judgment of a programming language, when other factors, such as ease of use, absence of quirks and gotchas, interoperability with other software for related tasks and user familiarity, might have greater practical significance to daily applications. Again, Matlab might be just great on these other considerations, too, for a given user.

[Randy Chugh]

Thanks, all. I understand that vectorization can speed things up, but that's true in Matlab also and therefore does not address my issue. And I agree that speed is not the only issue, but it is certainly a major one. Mike, to your point, it might be true that Mata is faster than Matlab in a less contrived example - e.g., estimating a maximum likelihood function - but for now, that's a conjecture. I will try to set up a less contrived apples-to-apples comparison, time permitting and will post results when I am able to get to that. Given that Matlab is probably the most popular comparable commercial product to Mata, I'm surprised StataCorp has not provided results of its own horserace between the two.

Lastly, MIke, thanks for the bag-of-tricks link above, very helpful!

Thanks again,
Randy

[daniel klein]

Thanks, all. I understand that vectorization can speed things up, but that's true in Matlab also and therefore does not address my issue.

Have you actually timed a vectorized version of your code in MATLAB? If there's no speedup, it's pretty likely that the MATLAB compiler vectorizes your explicit loop. In that case, vectorization (or the lack thereof in Mata's compiler) does address your issue.

[Randy Chugh]

Yes, I have. Vectorized MATLAB is much faster than for-loop MATLAB. As noted, I will prepare - time permitting - a better speed comparison with a program focused more on linear algebra and less on looping speed. Thanks.