p-values and BIC show the opposite results in model selection

[Min Zhang]

Hello Statalists,

I am using poisson regression for a three-way count dataset (5*5*5) and p-value and BIC index indicate the different results in model selection
Model 1 is nested in Model 2. Model 1 has the p-value (<.05), which means that it does not fit the data, but its BIC values is lower (more negative) than that of Model 2. The p-values of Model 2 is larger than .05, which means that this model fits the data well. Deviance is reduced in Model 2 and Likelihood test prefers Model 2 over Model 1.


A paper that is relevant to my research topic shows the similar results and prefers Model 2 over Model 1.The authors note that BIC is used as a model selection criterion when the sample size so large that no model would fit the data. The Ns (before I collapsed the data into the three-way count format) range from 1296 to 5832.However, I may need to note, if relevant, that the authors only presents these results in their working paper version and in the publication paper, they used a different sampling method and does not report the model selection results.

My codes are as following. The variables a, b, c are 5-class categorical variables. I am using Stata 12 SE.

.  // Model 1
. quietly poisson count i.a##i.b i.b##i.c 

. estat gof

         Deviance goodness-of-fit =  116.6563
         Prob > chi2(80)          =    0.0047

         Pearson goodness-of-fit  =   118.975
         Prob > chi2(80)          =    0.0031

. quietly fitstat, saving(M1)

. 
. // Model 2
. quietly poisson count  i.a##i.b i.b##i.c i.a##i.c

. estat gof

         Deviance goodness-of-fit =  58.46142
         Prob > chi2(64)          =    0.6718

         Pearson goodness-of-fit  =  58.84949
         Prob > chi2(64)          =    0.6586

. fitstat, using(M1)

Measures of Fit for poisson of count

                             Current            Saved       Difference
Model:                       poisson          poisson
N:                               125              125                0
Log-Lik Intercept Only:    -1673.722        -1673.722            0.000
Log-Lik Full Model:         -303.837         -332.934           29.097
D:                           607.673(34)      665.868(59)      -58.195(-25)
LR:                         2739.770(60)     2681.575(44)       58.195(16)
Prob > LR:                     0.000            0.000            0.000
McFadden's R2:                 0.818            0.801            0.017
McFadden's Adj R2:             0.764            0.762            0.002
Maximum Likelihood R2:         1.000            1.000            0.000
Cragg & Uhler's R2:            1.000            1.000            0.000
AIC:                           6.317            6.383           -0.066
AIC*n:                       789.673          797.868           -8.195
BIC:                         443.511          380.998           62.513
BIC':                      -2450.071        -2469.130           19.058

Difference of   19.058 in BIC' provides very strong support for saved model.

I am confused at the moment. Which model does fit the model better? Whether the interaction term (a##c) that is added in Model 2 is statistically meaningful?
My guess is that Model 2 fits the data better than does Model 1, because they are nested and likelihood test prefers Model 2. The interaction term (a##c) is statistically significant and should be taken into account in models, but its magnitude of the effect might be a bit limited.
I am not sure if my guess is right or not and how convincing it is.

[Maarten Buis]

That is real research: there is no absolute way to say which model is "best". First question you need to ask yourself: Are you substantively interested in the interaction? If yes, your model choice is easy: You should always include the interaction regardless of p-values or BICs, AICs or similar statistics. If no, the anser is more complex: A model is by definition a simplification of reality, and s aimplification is just "wrong is some useful way", so you have a trade-off how much "wrong" are you willing to accept for how much "simpleness". That depends to a large extend on the exact research question, but also on things like your own subjective taste (and that of important members of your audience, like reviewers, supervisors, etc.). An important step is not looking at p-values but at strength of the effect. Just substantively interpret your parameter. That will allow you to make an informed decision on whether this paremater is important for you or not.

[Joao Santos Silva]

In addition to Maarten's excellent advice, I suggest that you also ask yourself whether you really need the data to be Poisson. That is, do you need to compute probabilities of events, or do you just want to estimate a conditional expectation? In the second case, the criteria you are considering are irrelevant.

Joao

[wanhaiyou]

That is real research: there is no absolute way to say which model is "best". First question you need to ask yourself: Are you substantively interested in the interaction? If yes, your model choice is easy: You should always include the interaction regardless of p-values or BICs, AICs or similar statistics. If no, the anser is more complex: A model is by definition a simplification of reality, and s aimplification is just "wrong is some useful way", so you have a trade-off how much "wrong" are you willing to accept for how much "simpleness". That depends to a large extend on the exact research question, but also on things like your own subjective taste (and that of important members of your audience, like reviewers, supervisors, etc.). An important step is not looking at p-values but at strength of the effect. Just substantively interpret your parameter. That will allow you to make an informed decision on whether this paremater is important for you or not.

Hi Maarten
What does the sentence mean ?-"An important step is not looking at p-values but at strength of the effect"
What is "the strength of the effect"? Could you please give me an example? Thanks very much.

wanhaiyou