Contradicting significance in complete pooled vs partial-pooling

[Luca Giangregorio]

Hi all,

I'm working on nested structure data with individuals nested in countries (I may also define a more detailed structure with individuals nested in country-years nested in countries).
To analyze these data, I started with a complete pooled-OLS, simply clustering the standard errors at the country (or country-wave) level.

 reg isei_r c.maxisei_cntr##ib2.ineq_group i.gender age i.emp_status ib2.gdp_group beta_coeffadjMax [pw=dweight] , vce(clu cwave)  
*Same if I cluster for country only

However, when I turn to the partial-pooling to take into account both within and between variation, I lost significance on mine main predictors.

 mixed isei_r c.maxisei_cntr##ib2.ineq_group i.gender age i.emp_status ib2.gdp_group beta_coeffadjMax [pw=dweight] || country : maxisei_cntr, cov(unstr)

What could be the reason for this change in significance? Following Gelman & Hill (2007) I know that when the standard deviation of the group-level tends to 0, the model gains very little from the between-effect. In my case the ICC is also very low (0.012) and the std deviations at the individual level are 0.049 for the slope and 1.7 for the intercept, while the residual std deviation is 15.45.
Furthermore, comparing the residual variance of the multilevel with the residual variance of the complete-pooled they are very close. Is this suggests that the hierarchical model does not add basically anything to the complete pooled OLS?

Thanks for the support

[Clyde Schechter]

The American Statistical Association has recommended that the concept of statistical significance be abandoned. See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.

Even if you still want to focus on statistical significance as if it were meaningful and useful, as Gelman often reminds us, the difference between statistically significant and not statistically significant is, itself, not statistically significant. Not only is the difference between p = 0.04 and p = 0.06 meaningless, so is the difference between p = 0.1 and p = 0.01. The better way to compare the findings of the two models is to just look at the regression coefficients themselves. Have they changed by an amount that is meaningful in the real world? If not, just forget about the p-values and move on.

The low ICC does suggest that your multi-level model doesn't add much. Nevertheless, you should not disregard its results simply because you like the results of the OLS better. That's not science. Again, if, counter to my advice, you still wish to remain in the paradigm of significance testing, you can compare the models using a likelihood ratio test. Or, what is more honest, in my view, is to report both results.

[Carlo Lazzaro]

Luca:
as an aside to Clyde's helpful advice, the trivial comment is that you ran two (very) different regression models and you obtained different p-values and 95CIs.
So far so good.
Now, a more substantive issue rests on which model gives a fairer and truer view of the data generating process you're investigating: perhaps, the literature in your research field can give you some inputs.

[Luca Giangregorio]

Dear Clyde,
thanks for your reply. I'm agree with you and I'm not a "fetishist" of p-value and significance. My concern was mostly on understanding the dynamics behind this change and try to figure out what's happening behind the scene. Apart of the statistical significance itself.
Furthermore, I'm also agree on the low ICC that's not a sufficient argument to not proceed with a hierarchical model. Indeed, I was more focused on the residual variance which is basically the same in the two models. My doubt is basically more focused on understanding the reasons that may generate differences in the standard errors and if in case of same explanatory power of the model is it better to go with the more parsimonious one.

Thanks a lot again!

[Luca Giangregorio]

Luca:
as an aside to Clyde's helpful advice, the trivial comment is that you ran two (very) different regression models and you obtained different p-values and 95CIs.
So far so good.
Now, a more substantive issue rests on which model gives a fairer and truer view of the data generating process you're investigating: perhaps, the literature in your research field can give you some inputs.

Thanks Carlo for your comment as well.
ideally I would think the hierarchical model to better represent the data, even in logical terms. Furthermore, since I would like to grasp the cross-country differences, it may also be more intuitive to opt for a multilevel. However, focusing on the more econometric aspect, I was just trying to understand why I have very different standard errors if the multilevel tend to be a complete pooled model in the case the group residual variance tends to 0. But yes, probably as Clyde mentioned, could be useful to present both results.

[Carlo Lazzaro]

Luca:
if you actually have a nested design, you should go -mixed- as your first choice, no matter what Stata gives you back.
Then, if your results does not show evidence of random intercept (or random intercept and random slope), you may explain why an OLS can do with your dataset.