AIC veersus BIC fitstat

[Federica Di Marcantonio]

Dears,

I using the command fitstat, diff to compare different models. In particular I am comparing a simple model (logistic y x) with a more complete one (adding to the previous one additional explanatory variables The results I get are reported in the following table. But I am confused on the contradictory results I get by looking at AIC and BIC.
Generally the model with the smaller BIC is preferred. Similarly, all else being equal, the model with the smaller AIC is considered the better fitting model.
According to the results presented in the table BIC supports the choice of the "reduced model" (the one with only one expl. var) whilst AIC the other (the full version).
Can anyone help me to understand the reason(s) behind these different results and interpret them in a correct way.

Thanks

Federica

	Current (M1 reduced)	saved (M1 full)	Difference
Log-likelihood
Model	-357.2	-310.6	-46.6
Intercept-only	-390.6	-390.6	0.0
Chi-square
D (df=888/871/17)	714.4	621.1	93.2
LR (df=1/18/-17)	66.912	160.152	-93.240
p-value	0.000	0.000	0.000
R2
McFadden	0.086	0.205	-0.119
McFadden (adjusted)	0.081	0.156	-0.076
Cox-Snell/ML	0.072	0.165	-0.092
Cragg-Uhler/Nagelkerke	0.124	0.282	-0.158
Efron	0.073	0.188	-0.115
Tjur's D	0.076	0.190	-0.114
Count	0.840	0.854	-0.013
Count (adjusted)	0.000	0.085	-0.085
IC
AIC	718.4	659.1	59.2
AIC divided by N	0.807	0.741	0.067
BIC (df=2/19/-17)	728.0	750.2	-22.2
N

Difference of 22.211 in BIC provides very strong support for current model

[German Rodriguez]

These criteria have different penalties for model complexity. AIC imposes a penalty of 2 per parameter, while BIC uses log(N). Looks like in your case N=890 so log(N)=6.79, a bigger penalty. I am guessing your models have 2 and 19 parameters. Using the log-likelihoods you list, this code reproduces your results:

. scalar N = 890

. di "AIC", -2*(-357.2) + 2*2, -2*(-310.6) + 2*19
AIC 718.4 659.2

. di "BIC", -2*(-357.2) + log(N)*2, -2*(-310.6) + log(N)*19
BIC 727.98244 750.23321

As you can see, the only difference is the use of a higher penalty per parameter in BIC

[Richard Williams]

There is nothing that says BIC and AIC have to agree (if they always did I guess we wouldn't use two measures.) Nor do you have to make the model choice just based on these two measures. You can note that evidence from fit measures is mixed but go with the one you find most sensible.

[Brad Anderson]

As Richard said, different fit statistics don't always agree. My experience is that they almost never agree. I'm certainly not an expert on the relative merits of BIC vs AIC. I've become somewhat interested in using the Bayes factor as an alternative to null hypothesis testing since I find standard null hypothesis testing, at least as implemented in practice, very limiting in most circumstances. Of course, it is standard accepted practice in many fields and bucking the trend is often challenging. I once had an American Medical Association journal asking for exact p-values when I had reported bias-corrected and accelerated confidence intervals. They actually had trouble grasping the idea that exact p-values weren't available. But I digress. At any rate, an approximate Bayes factor comparing two models can be easily derived from the BIC statistics. Some of the folks associated with Mplus have published simulations comparing various fit statistics and BIC seems to usually be better than AIC. Of course, that only extends to the kinds of problems simulated. In my experience, AIC tends to suggest overly complex models. All that said I don't think there's any real consensus that one fit statistic is necessarily better than the other, so my advice would be to consider them both but ultimately choose a model based on theoretical/conceptual rationale rather than purely on the basis of either AIC or BIC.

[Federica Di Marcantonio]

Thank you for your insights. I am not familiar with Bayes factor for comparing two models and how can be retrieved from BIC. I have to clarify that this test was not for choosing the model but rather to confirm that our full model was "better"than the simple one. I have to say that I do not feel completely comfortable in reporting this table (this was my intention) and go for the more "convenient" test. including these contradictory evidence is not helping in explaining what I want to show, and the justification of higher penalty per parameter in BIC might not be very strong...anyway thanks

[Brad Anderson]

Thank you for your insights. I am not familiar with Bayes factor for comparing two models and how can be retrieved from BIC. I have to clarify that this test was not for choosing the model but rather to confirm that our full model was "better"than the simple one. I have to say that I do not feel completely comfortable in reporting this table (this was my intention) and go for the more "convenient" test. including these contradictory evidence is not helping in explaining what I want to show, and the justification of higher penalty per parameter in BIC might not be very strong...anyway thanks

Well if you want an introduction to the Bayes factor you might want to take a look at "A practical solution to the pervasive problems of p-values by Eric-Jan Wagenmakers. It's a 2007 article in the Psychonomic Bulletin & Review. Issue 14(5), 779-804. I'm wondering what you mean by full model versus the sample model? That concerned me a bit in that you can only compare models for a common sample using the same estimator. It can't be used to compare different samples or different estimation procedures. The usually recommendations for comparing the evidence provided by different BICs essentially gives the same conclusion as the actual Bayes factor (which is actually only approximated using BIC statistics. The Bayes factor basically gives the probability that one model is more likely than another, given the observed data. There's tons of literature outlining the problems associated with null hypothesis testing.