Calculating standard errors when "variance matrix is nonsymmetric or highly singular"

[Anya Fedyk]

Dear smart people.

I encountered this problem several times but now I want to solve it for good. When I'm doing regressions, typically with 2 fixed effects, sometimes I get the warning "variance matrix is nonsymmetric or highly singular".

Coefficients are computed but there are no standard errors. here comes the weird thing:

with most regressions, after I execute the exact same command several times, it works in one of the cases. For a few cases I have tried >20times and it never worked.

Regressions take the form

areg y x1 x2 .... i.smallGroup if ....., absorb(largeGroup) robust cluster(smallGroup OR another)

system is Stata SE 13.1, windows 7prof, 32bit

This should not be a mathematical problem but a software issue?

If anyone could tell me a reliable solution, my forever gratitude would be certain!
AF

[Anya Fedyk]

I know that this typically happens when there are clusters with only a few observations. If I drop clusters with <25 obs it basically works, if I keep clusters <5obs it typically does not work. What irritates me is that when I do the exact same thing, sometimes standard errors are there, sometimes not.

I was wondering whether someone can clarify me where this (as it appears to me) arbitrariness comes from and how I determine the cutfoff for the number of observations needed within one cluster depending on the regression setup. This is important because I want to produce many columns and outreg them all without having to check whether there are standard errors, or not. Thank you!

[Mark Schaffer]

Anya - a good, short explanation of what can cause you to get a singular variance matrix is in the online help in help j_robustsingular.

But the interesting question for me is why Stata is sometimes able to post a singular variance matrix and report regression results (which is where the link above comes from) and why sometimes it refuses, reports "Warning: variance matrix is nonsymmetric or highly singular" and then exits with an error and without posting any results.

Anybody have any ideas on this?

[Mark Schaffer]

I tried using Anya's description of her problem using areg with grunfeld data, but I can only get the "help j_robustsingular" version of the error (click on the missing F stat in the upper RH corner of the areg output).

webuse grunfeld, clear
gen smallgroup = round(time/3)
areg invest mvalue i.smallgroup, absorb(company) cluster(smallgroup)

My own experience is that the "Warning: variance matrix is nonsymmetric or highly singular" is rarely seen in the wild and I haven't been able to come up with a replicable version using an official Stata command and openly available data. Can anybody else manage to do this?

[Anya Fedyk]

thank you Mark. I followed the help you mentioned where is says

"
Are any standard errors missing?

If any standard errors are reported as dots, something is wrong with your model: one or more
coefficients could not be estimated in the normal statistical sense. You need to address that problem
and ignore the rest of this discussion.

"

well, in the cases I mentioned ALL standard errors are reported as dots.

Furthermore, in an exmaple I am currently looking at there are four coefficients of i.smallgroup that are reported as "0". could this have to do with the problem? probably not because when I drop some more clusters with, say less than 80obs, (it wasn't working with at least 50 in my example), then there are standard errors everywhere, but the four i.smallgroup coefficients are still 0 (and have no SEs reported)

Do I have to check all clusters before I run such a regression and what do I exactly look for? if there is any variation?

I am sorry but the data I work with is confidential so I cannot upload it.

Thank you!

[Mark Schaffer]

Anna - I think all the SEs are dots because Stata is refusing to post the entire estimated VCV. What I can't understand is why Stata sometimes, but only sometimes, completely refuses to post a singular VCV. The areg estimation I posted is an example of a VCV that's singular and yet Stata posted it. ("post" = "include in the regression results")

Am I right in reading your original description that the problem is not 100% replicable? That is, you can run exactly the same regression on exactly the same data several times in succession, and sometimes Stata exits with the "variance matrix highly singular" error and sometimes it is able to post the variance matrix without this error? That's very odd. It's possible that non-stable sorting could be behind it - there was a discussion about this in connection with xtreg (a close relative of areg) here:

http://www.statalist.org/forums/foru...command-is-run

My only suggestion is to contact Stata Tech Support. (And let us know what they say!)

[Joao Santos Silva]

Dear Anya,

Are you by any chance doing something that sorts your data? That can explain why it does not happen all the time.

All the best,

Joao

[Joselin Segovia]

Dear Anya,

Are you by any chance doing something that sorts your data? That can explain why it does not happen all the time.

All the best,

Joao

Dear Joao,

Could you please help me? I get the same error than Anya, the difference is I'm estimating a demand system of 4 goods through the AIDS model and I get the message "Warning: variance matrix is nonsymmetric or highly singular", then the output is shown with only the coefficients. Any help would be much appreciated.

Joselin

[Joao Santos Silva]

Dear Joselin,

There are a number of possible causes for this. The most likely is that you have singletons (dummies with just one observation equal to 1), but the problem may have other causes. What kind of covariance matrix are you estimating?

Joao