Too large a -matsize- degrades performance (Was: "-clogit- running *grossly* slower under v.13 MP2 vs. v. 12 IC)

[Mike Lacy]

Greetings,

I'm posting the wrapup on this discussion under a new subject, as I think it will bring useful information to more people than an apparently esoteric issue with -clogit-. For those interested, all the detail is in the original posting/conversation.

Summary: Per the old subject, I started by complaining of -clogit- very slowly under v. 13 MP2, and I thought the problem was an MP2 vs. IC problem or a Stata version problem. It turned out, as Hua Peng of StataCorp explained at the posting above, that the issue was that I had a very large maximum matrix size set in MP2, 11,000, vs. the 800 max that is possible in IC. When I reset to a small -matsize- (say 1000), the performance problem went away. The problem is that a large -matsize-, even in a machine with apparently large amounts of memory available, can in some cases cause problems with the machine's efficiently using main memory and cache memory.

Using a large matsize that is not needed is not harmless, which was news to me. In the case at hand, it unnecessarily slowed down -clogit- by a factor of 30.

Regards, Mike

[Richard Williams]

Thanks Mike. Seems like this would be a good choice for a FAQ.

[Ariel Karlinsky]

Sorry to bump an old thread but Iv'e had a similar issue just now. estimations that usually take about 20 minutes now took about 40. I quickly recognized that the issue was that for a previous estimation which required a larger matsize - I set it to 2000 instead of 400. Everything was slow to a crawl, extremely frustrating. Thanks god I found this thread, I reduced matsize back to 400 and everything's back to normal.

If I'm not mistaken this issue is STILL NOT in the offical FAQ of stata, which is a darn shame.