Predictions using Fixed Effects conditional logit

[Stephanie Atala]

I have a panel data for 453 groups over a period of 6 years. The objective is to use the previous year properties (predictors) of a group to predict an outcome of 0/1. After fitting a conditional fixed effects logit model I got a McFadden's R-square of 0.243. I calculated the predicted probabilities using Hosmer, Lemeshow, and Sturdivant 2013 eq. 7.4 shown in the picture attached. After getting these predicted probabilities, I get a brier score of 0.32. However, If I calculate the brier using simple prediction based on historical data, i.e. I only look at the number of positive outcomes in the past years without taking into account the predictors of the model, I get a better brier score of 0.22. Is this considered normal? Is it a property of the logit model to have limited predictability?

Any insight is appreciated.

[Joao Santos Silva]

You cannot do any meaningful predictions with the FE logit; see here (specially #6).

Best wishes,

Joao

[Stephanie Atala]

Thank you for your reply Joao Santos Silva . So would you say the only takeaway from my model would be the odds ratio? I am trying to see how will I be able to validate my model especially when I want to write the paper about it and get reviews. Like how can I say that my model is a good valid model?

Thanks again, I appreciate it!

[Joao Santos Silva]

You can do a RESET-type test to check the specification. Note that in your initial post you say you want to use your model for prediction; for that you cannot use models with fixed effects.

Best wishes,

Joao

[Stephanie Atala]

Joao Santos Silva Can you please further explain how to do the RESET-type test and what does it mean in terms of interpretation?

Thanks again!

[Joao Santos Silva]

Dear Stephanie,

Below I provide an example of how to do it. The null is that the model is correctly specified, in the example below the null is not rejected and therefore the test provides no evidence against the model.

Best wishes,

Joao

webuse union
xtlogit union age grade i.not_smsa south##c.year, fe
predict xb, xb
g xb2=xb^2
g xb3=xb^3
xtlogit union age grade i.not_smsa south##c.year xb2 xb3, fe
*Perform the RESET test
test xb2 xb3

[Stephanie Atala]

Joao Santos Silva

Dear Joao,

I went ahead and performed the reset test for xb2 and xb3 and got a Prob > chi2 = 0.0016 as output. Therefore, we fail to reject the null hypothesis. To overcome this, should I do transformations to my variables? or add interaction terms for example?

Thanks again!

Stephanie

[Joao Santos Silva]

To be precise, you reject the null, right? Yes, interactions, and transformations are good ideas.

Best wishes,

Joao