Goodness of fit test for log-binomial model with survey data

[Tanvir Hasan]

Dear Stata Users,

I ran a log-binomial regression model with survey data to estimate relative risk of childhood stunting. The Stata output with svy command doesn’t report goodness of fit statistic, nor even AIC or BIC. Can anyone kindly suggest how to perform a global test of goodness of fit after fitting a log-binomial regression model?

I am trying with the following glm model
xi:svy: glm stunted i.wrk i.region i.div i.bmi, fam (bin) link (log) eform difficult

Thank you in advance for your kind help.

Kind regards,
Tanvir Hasan

[Richard Williams]

This doesn't answer your question, but unless you are using an old version of Stata you usually don't want to use the xi: prefix anymore. Stata now has built-in support for factor variables. Type

help fvvarlist

for details.

[Maarten Buis]

What is the null hypothesis you want a "global test of goodness of fit" to test? Statistical tests cannot test a general hypothesis like "the model fits", it has to be more specific like "all coeffcients except the constant equal 0" like the F-test after linear regression.

[Richard Williams]

Stats like BIC and AIC are not legit with svyset data, because assumption of ML estimation are violated. Nonetheless, if you want to cheat, you can get these stats out. This might be ok if you are just looking for some kind of suggestive ballpark measure. For example, you can do

webuse nhanes2f, clear
svy: reg health weight
* Next command will not work
estat ic
reg health weight [pw = finalwgt]
estat ic
est store m1
reg health weight height [pw = finalwgt]
estat ic
est store m2
* Next command will not work
lrtest m1 m2, stats 
lrtest m1 m2, stats force

Also, if you type

help svy: estat

you see some postestimation commands you can use. There is an estat gof command that works after things like logit, but it doesn't sound like that is what you want.

[Tanvir Hasan]

Dear Marteen and Richard,

Thank you very much for your kind feedback.

Maarten, I have performed the joint hypothesis test (all coefficients except the constant equal 0) using the test command in Stata. My problem is with the goodness of fit of the resulting log-binomial model.

Richard, you are right that the estat gof command can be used to check for the goodness of fit for a logistic regression, but I would like to know if there is any Stata ado-command available for log-binomial regression. Please note that I am using Stata 12.

Looking forward to your reply.

Kind regards,
Tanvir Hasan

[Maarten Buis]

[QUOTE=Tanvir Hasan;n31734]
Maarten, I have performed the joint hypothesis test (all coefficients except the constant equal 0) using the test command in Stata. My problem is with the goodness of fit of the resulting log-binomial model.
/QUOTE]

That does not answer my question. To repeat, what null-hypothesis do you want your "goodness of fit test" to test?

Remember, the answer can not be that the model "fits", as that is too ambiguous. There can be many, somtimes conflicting, measures of fit, and none of them tell the whole story. We obviously cannot and should not decide for you what hypothesis you want to test. The easiest for us (and probably you) to identify what test you want to perform is to specify the null-hypothesis.

[Richard Williams]

I suspect Tanvir is hoping somebody has already figured out the answers to Maarten's questions and figured out how to program them. There is an -estat gof- command that works after logit, probit, sem, and poisson. The Hosmer-Lemeshow goodness-of-fit test after logit has been widely criticized, but some of the other tests make good sense to me. I suggest Tanvir Google

goodness of fit test for log binomial model

and see if any of the resources listed seem helpful. e.g.

http://www.ncbi.nlm.nih.gov/pubmed/16544809

[Rich Goldstein]

I'm not completely sure what Richard W means about the Hosmer-Lemeshow test; I am familiar with issues surrounding "too much" power for the test for sample sizes greater than about 1,000; a recent article helps with this: Paul, P, Pennell, ML and Lemeshow, S (2013), "Standardizing the Power of the Hosmer-Lemeshow goodness of fit test in large data sets", _Statistics in Medicine_, 32: 67-80

[Richard Williams]

Here is Paul Allison's critique of H-L:

http://www.statisticalhorizons.com/hosmer-lemeshow

Also see

http://www.statisticalhorizons.com/a...-lemeshow-test

http://www.statisticalhorizons.com/a...tic-regression

[Rich Goldstein]

thanks - I note that the paper I cited just above (#8) deals with the issue of choosing the number of groups