Dear Statalist members,
I am struggling to decide how to compare two distinct models (with and without a certain variable) for a probit model. I am aware that theory should guide my decision more than econometrics tests, but in this case this is a new field of study and there is a lot that is not standard.
In order to decide about the variable that I am concerned with, I have considered / used: likelihood-ratio, Wald test, AIC / BIC, Pearson, and sensitivity/ specificity measures. (I am assuming that in this case I can consider tests of model comparison and tests of goodness of fit as similar, please correct me if I am wrong).
I have 700 observations in my dataset, and my model is the following model:
P(part=1) = F(β1*d + β2*c1 + β3*d*c + β4*lg + β5*pv1 + β6*td + β7*lf + β8*g)
The variables we are interested in are:
. d, a dummy variable indicating some characteristic of the firm
. c1, a continuous variable indicating some characteristic of the firm
. lg, a continuous control variable which is highly correlated with c1, but not at all correlated with part.
(The others are all controls that do not matter for this discussion.)
The analysis focuses on the interaction term d##c.c1
lg is highly correlated with c1, but not at all with part (which, for me, is an indication that it will not help explaining the dependent variable, please correct me if I am wrong).
So I run my model with variable lg and the corresponding tests.
And then I run the model without lg.
Finally, I compare the two models using a likelihood ratio test.
Bottomline, I have two questions:
1. Should I report all the tests? Should I do something else?
2. Does the fact that the wald test does not run after a regression with i., c. and ## mean that I should not run it?
Best,
MM
I am struggling to decide how to compare two distinct models (with and without a certain variable) for a probit model. I am aware that theory should guide my decision more than econometrics tests, but in this case this is a new field of study and there is a lot that is not standard.
In order to decide about the variable that I am concerned with, I have considered / used: likelihood-ratio, Wald test, AIC / BIC, Pearson, and sensitivity/ specificity measures. (I am assuming that in this case I can consider tests of model comparison and tests of goodness of fit as similar, please correct me if I am wrong).
I have 700 observations in my dataset, and my model is the following model:
P(part=1) = F(β1*d + β2*c1 + β3*d*c + β4*lg + β5*pv1 + β6*td + β7*lf + β8*g)
The variables we are interested in are:
. d, a dummy variable indicating some characteristic of the firm
. c1, a continuous variable indicating some characteristic of the firm
. lg, a continuous control variable which is highly correlated with c1, but not at all correlated with part.
(The others are all controls that do not matter for this discussion.)
The analysis focuses on the interaction term d##c.c1
lg is highly correlated with c1, but not at all with part (which, for me, is an indication that it will not help explaining the dependent variable, please correct me if I am wrong).
Code:
. pwcorr lg c1 part d, star(0.01)
lg c1 part
c1 0.2560*
part -0.0258 0.3834*
d 0.0474 -0.7392* -0.4537*
Code:
. probit part i.d##c.c1 lg lf i.d##c.td i.d##c.pv1 b3.g
Iteration 0: log likelihood = -472.92546
Iteration 1: log likelihood = -374.7233
Iteration 2: log likelihood = -373.6186
Iteration 3: log likelihood = -373.61627
Iteration 4: log likelihood = -373.61627
Probit regression Number of obs = 707
LR chi2(16) = 198.62
Prob > chi2 = 0.0000
Log likelihood = -373.61627 Pseudo R2 = 0.2100
-------------------------------------------------------------------------------
part | Coef. Std. Err. z P>|z| [95% Conf. Interval]
--------------+----------------------------------------------------------------
d |
Unsuccessful | -2.956323 1.487472 -1.99 0.047 -5.871716 -.0409311
c1 | -.1150382 .1660536 -0.69 0.488 -.4404973 .2104208
|
d#c.c1 |
Unsuccessful | .2103382 .1634254 1.29 0.198 -.1099697 .5306461
|
lg | -.0430944 .1516665 -0.28 0.776 -.3403552 .2541665
lf | .0562705 .0399928 1.41 0.159 -.022114 .134655
td | -.0564933 .020995 -2.69 0.007 -.0976428 -.0153438
|
d#c.td |
Unsuccessful | .0113093 .0392728 0.29 0.773 -.0656641 .0882827
|
pv1 | .0804259 .0369143 2.18 0.029 .0080752 .1527766
|
d#c.pv1 |
Unsuccessful | .0442204 .0716646 0.62 0.537 -.0962396 .1846803
|
g |
2 | -.7994469 .2561994 -3.12 0.002 -1.301588 -.2973054
4 | -.0106623 .147667 -0.07 0.942 -.3000842 .2787597
5 | -.3968741 .1742177 -2.28 0.023 -.7383345 -.0554137
6 | -.3198792 .3833424 -0.83 0.404 -1.071217 .4314581
7 | -.0111446 .1920815 -0.06 0.954 -.3876175 .3653283
8 | -.2505498 .3106359 -0.81 0.420 -.859385 .3582853
9 | .2896699 .2191254 1.32 0.186 -.1398081 .7191478
|
_cons | 1.853166 .9211391 2.01 0.044 .0477669 3.658566
-------------------------------------------------------------------------------
. estat class
Probit model for part
-------- True --------
Classified | D ~D | Total
-----------+--------------------------+-----------
+ | 377 123 | 500
- | 54 153 | 207
-----------+--------------------------+-----------
Total | 431 276 | 707
Classified + if predicted Pr(D) >= .5
True D defined as part != 0
--------------------------------------------------
Sensitivity Pr( +| D) 87.47%
Specificity Pr( -|~D) 55.43%
Positive predictive value Pr( D| +) 75.40%
Negative predictive value Pr(~D| -) 73.91%
--------------------------------------------------
False + rate for true ~D Pr( +|~D) 44.57%
False - rate for true D Pr( -| D) 12.53%
False + rate for classified + Pr(~D| +) 24.60%
False - rate for classified - Pr( D| -) 26.09%
--------------------------------------------------
Correctly classified 74.96%
--------------------------------------------------
. estat ic
Akaike's information criterion and Bayesian information criterion
Model Obs ll(null) ll(model) df AIC BIC
. 707 -472.9255 -373.6163 17 781.2325 858.7701
Probit model for part, goodness-of-fit test
number of observations = 707
number of covariate patterns = 707
Pearson chi2(690) = 702.39
Prob > chi2 = 0.3634
probit part d c1 d#c.c1 lg lf td i.d#c.td pv1 i.d#c.pv1 g
. test d c1 lg lf td pv1 g
( 1) [part]d = 0
( 2) [part]c1 = 0
( 3) [part]lg = 0
( 4) [part]lf = 0
( 5) [part]td = 0
( 6) [part]pv1 = 0
( 7) [part]g = 0
chi2( 7) = 59.79
Prob > chi2 = 0.0000
Code:
. probit part i.d##c.c1 lf i.d##c.td i.d##c.pv1 b3.g
Iteration 0: log likelihood = -472.92546
Iteration 1: log likelihood = -374.74916
Iteration 2: log likelihood = -373.65908
Iteration 3: log likelihood = -373.65662
Iteration 4: log likelihood = -373.65662
Probit regression Number of obs = 707
LR chi2(15) = 198.54
Prob > chi2 = 0.0000
Log likelihood = -373.65662 Pseudo R2 = 0.2099
-------------------------------------------------------------------------------
part | Coef. Std. Err. z P>|z| [95% Conf. Interval]
--------------+----------------------------------------------------------------
d |
Unsuccessful | -3.274932 .9812371 -3.34 0.001 -5.198122 -1.351743
c1 | -.1516272 .1052039 -1.44 0.150 -.3578229 .0545686
|
d#c.c1 |
Unsuccessful | .2440799 .1126263 2.17 0.030 .0233363 .4648234
|
lf | .0564968 .0399777 1.41 0.158 -.021858 .1348516
td | -.056693 .020982 -2.70 0.007 -.097817 -.0155691
|
d#c.td |
Unsuccessful | .0105463 .0391434 0.27 0.788 -.0661734 .087266
|
pv1 | .0804785 .0369321 2.18 0.029 .0080929 .1528642
|
d#c.pv1 |
Unsuccessful | .044813 .071604 0.63 0.531 -.0955282 .1851541
|
g |
2 | -.7971488 .2559817 -3.11 0.002 -1.298864 -.2954339
4 | -.0075489 .1472343 -0.05 0.959 -.2961228 .2810251
5 | -.3966199 .1742109 -2.28 0.023 -.738067 -.0551729
6 | -.3135455 .3820834 -0.82 0.412 -1.062415 .4353242
7 | -.0109047 .1921097 -0.06 0.955 -.3874329 .3656234
8 | -.2484771 .3108681 -0.80 0.424 -.8577674 .3608132
9 | .2845485 .2182287 1.30 0.192 -.1431719 .712269
|
_cons | 1.798677 .9014511 2.00 0.046 .0318656 3.565489
-------------------------------------------------------------------------------
. estat class
Probit model for part
-------- True --------
Classified | D ~D | Total
-----------+--------------------------+-----------
+ | 377 123 | 500
- | 54 153 | 207
-----------+--------------------------+-----------
Total | 431 276 | 707
Classified + if predicted Pr(D) >= .5
True D defined as part != 0
--------------------------------------------------
Sensitivity Pr( +| D) 87.47%
Specificity Pr( -|~D) 55.43%
Positive predictive value Pr( D| +) 75.40%
Negative predictive value Pr(~D| -) 73.91%
--------------------------------------------------
False + rate for true ~D Pr( +|~D) 44.57%
False - rate for true D Pr( -| D) 12.53%
False + rate for classified + Pr(~D| +) 24.60%
False - rate for classified - Pr( D| -) 26.09%
--------------------------------------------------
Correctly classified 74.96%
--------------------------------------------------
. estat ic
Akaike's information criterion and Bayesian information criterion
-----------------------------------------------------------------------------
Model | Obs ll(null) ll(model) df AIC BIC
-------------+---------------------------------------------------------------
. | 707 -472.9255 -373.6566 16 779.3132 852.2897
-----------------------------------------------------------------------------
Note: N=Obs used in calculating BIC; see [R] BIC note.
. estat gof
Probit model for part, goodness-of-fit test
number of observations = 707
number of covariate patterns = 707
Pearson chi2(691) = 702.36
Prob > chi2 = 0.3738
. quietly probit part d c1 d#c.c1 lg lf td i.d#c.td pv1 i.d#c.pv1 g
. test d c1 lg lf td pv1 g
( 1) [part]d = 0
( 2) [part]c1 = 0
( 3) [part]lg = 0
( 4) [part]lf = 0
( 5) [part]td = 0
( 6) [part]pv1 = 0
( 7) [part]g = 0
chi2( 7) = 59.79
Prob > chi2 = 0.0000
Code:
. quietly probit part i.d##c.c1 lf c.td c.pv1 b3.g . estimates store m1 . quietly probit part i.d##c.c1 lg lf c.td c.pv1 b3.g . estimates store m2 . lrtest m1 m2 Likelihood-ratio test LR chi2(1) = 0.08 (Assumption: m1 nested in m2) Prob > chi2 = 0.7793
Bottomline, I have two questions:
1. Should I report all the tests? Should I do something else?
2. Does the fact that the wald test does not run after a regression with i., c. and ## mean that I should not run it?
Best,
MM
Comment