Problem comparing my models - ovtest - RMSE - glm - regress

[Alejandro Mio]

Dear all,

I am using confidential data therefore I replicated an example after reading the FAQ.

I first searched solutions but couldn't find anything. I'm trying to model health costs and thanks to papers like Pr. Andrew M. Jones - "Models For Health Care" (https://www.york.ac.uk/media/economi...c/wp/10_01.pdf) I realized I should try different models and test them.

I used these models

 regress ln_costs covariates, robust

with the costs that I "logged" to try getting a normal distribution.

I used a Poisson regression

poisson costs covariates, vce(robust) irr (

but also

glm costs covariates, vce(robust) eform family(poisson) link(log)

hoping I could solve my issues as they return the same results but have different options.

I also used a

glm costs covariates, family(gamma) link(log) vce(robust) eform

I got results for these models and that's where my question comes, I can't manage to compare them using tests like RESET test, or comparing RMSE (like mentioned in the article I cited earlier)

My code example:

sysuse auto.dta, clear

gen ln_price = ln(price)

reg ln_price mpg weight rep78, robust

ovtest

di e(rmse)

glm price mpg weight rep78, family(gamma) link(log) robust eform

ovtest

ereturn list

I get a result for the RESET test or the RMSE when I use “regress” but I can't get them for the glm commands…

How to compute RESET tests for my glm or how can I compare my errors ? (If I can compare my errors, I would choose the model that has the best predictions of my costs)

But when I use the following, I can't use the

predict, residuals

command

glm costs covariates, family(poisson) link(log) vce(robust) eform
predict newvar, residuals

or

poisson costs covariates, vce(robust) irr
predict newvar, residuals

All these codes give me error "option residuals not allowed".

I saw on an other post the possiblity to use:

glm costs covariates, family(poisson) link(log) vce(robust) eform
predict newvar, res

I don't get the difference between using "res" and not "residuals" ... It seems like the same option, but one works and not the other ?
I read on that post that comparing errors for Poisson is hard... I used the

 help glm_postestimation

but It didn't help me understand.

Do you have a solution?

Thank you very much for the time you take on forums!

Alejandro

[Dung Le]

Hi Alejandro,

I think -ovtest- does not work because -glm- postestimation does not support -ovtest- command. An alternative way is to use -linktest-, which works a similar way as Ramsey’s RESET does, except it does not account for cubed or fourth-order terms. Of course, you can compute Ramsey's RESET test manually, but it requires a bit of work.

qui glm price mpg weight rep78, family(gamma) link(log) robust
    linktest

Based on the standard RMSE equation, then it can be computed as follows:

qui glm price mpg weight rep78, family(gamma) link(log) robust
    predict yhat, mu
    sum yhat
gen rmse = ((price-yhat)^2/e(N))^(1/2)

If you are modeling healthcare cost, another good option for you, along with the Poisson and -glm-, you may want to try to use two-part model command -twopm- written by Belotti, Deb, Manning, and Norton (Google two-part model in Stata to find their paper, I am unable to access to that paper due to poor internet quality here). The -twopm- is powerful in estimating healthcare costs.

[Alejandro Mio]

Hi Dung,

Thank you very much for your help. I will try all that.
Concerning the twopm, I saw it was mainly for health care costs with zeros whereas my data has no zeros.
Thank you again.