What do the insignificant regression estimates suggest?

[Seren Essex]

Hi there,

I run a logistic regression model to predict school exam success, N= 7493 and Df = 74. I find school type to be a significant predictor.

I now run two logistic regression models for male and female students separately. This is so that I can model the estimates by sex. Male, N= 6144 and female, N=1334. I simplify the models so Df=47.

I now find that school type is a significant predictor for the male model but not the female model.

Can I say that the effect of school type on school exam success is different for male and female students? Or could the insignificance in the female model be caused by small cell sizes?

How can I determine which interpretation of the insignificance is correct?

Thank you very much for your help.

Seren

[Ken Chui]

Whether it is under-powered or due to modification by sex is not something that can be "determined" by looking at a statistical output (not to mention that there isn't any output to be looked at). Stat is just part of the clues, hardly ever a complete hint.

1) I'd first make sure the sex-stratified analysis is pre-planned. Avoid slicing the dicing the data set and analyze its multiple facets in an ad hoc manner. That would grossly inflate the type I error rate.

2) Assume the sex-stratified analysis was pre-planned, I'd suggest first check if you even need to do that in the full sample: is there indeed a sex-by-school interaction? You can do something like this to check:

logit y i.sex##i.school
testparm sex#schoo

This test gives you a general idea if the association between outcome and school does depend on sex. If this is significant, then perhaps sex-stratified analysis is an appropriate next step. However, you can also get all the predicted outcomes, by sex and by school, from this model, I generally don't think stratified analysis very useful besides simpler to interpret.

3) Whether it is under-powered due to small cell size would require i) a proper power or effect size calculation, and ii) looking at the expected distribution. But generally, stratified analyses get people into this awkward situations. Some thought they are "drilling down" to the core of the issue, but in fact they are just repeatedly hurting their statistical power. For pre-planned stratified analyses, sample size and power determination should be checked within each stratum. This is usually not done in many observational studies. If this is your case I'd suggest re-compute the sample size and collect more data.

4) And whether it is an interaction between sex and school is not something that can be discerned from the output. It's important to have an actual mechanism (biological, social, etc.) to back that up. How could these school affect students' performance differently by sex? Are there potential biases in the system? Were boys taking classes that are more supported in terms teacher qualification or infrastructure? Did girls in the community bear household tasks and had to be absent more so than boys? Etc. It would also be useful to check the trend and magnitude of the two sets of school odds ratios between the two sexes. Do they trend together? Or do some schools have very opposite OR (e.g. much bigger than 1 for boy, and much smaller than 1 for girl, etc.)? That can also give some hint.