gsem estat- AIC BIC interpretation in large sample sizes

[Emmanuelle Belanger]

Hi everyone,

I am working on the validation of a latent measure with ordinal items (4-point Likert scales) and working with a very large sample size (400K). For this purpose, I am estimating gsem, ologit models and using the estat ic command to obtain the AIC and BIC statistics to compare competing factor structures. I've also estimated a polychoric correlation matrix and PCA. The problem is that factor loadings, internal consistency statistics (omega) and eigenvalues all strongly support a single factor model, but the BIC statistics are still significantly smaller for two-factor models. My question is, does anyone know if there are ways to correct for sample size when estimating the BIC or of discussion about appropriate benchmark values when examining this statistic in a large sample? I know that 10 is used as a threshold to indicate strong evidence of a better model for the Schwarz version of the BIC, but that doesn't seem appropriate with the output I am getting. Even using a 10% random sample (n= 40,827), I still obtain a two-factor model with a BIC that is 767 below the one-factor model (357,796 vs. 357,029). Note that the sample sizes are the same between models.

Most of the technical notes that I can find about BIC are not discussing gsem models specifically.
https://www.stata.com/manuals13/rbicnote.pdf#rBICnote
https://www.stata.com/statalist/arch.../msg00884.html

Thank you,
Emma

[Phil Bromiley]

You are comparing different criteria. Different criteria can give different answers. AIC often favors a different model than BIC for example.

[Philip Matthews]

A variation of the theme of comparing like with like: PCA does not produce factors — PCA rearranges the data into a simpler structure and ignores error as it does so. PCA may give hints about the possibility of recognising factors; but components aren't factors. Accounting for error is a key part of SEM and the testing of factor models. Thus it is not neessarily surprising if SEM indicates factorial structure(s) that do not match the pattern of components in PCA. In short, PCA summarises data; SEM with factors (latents) tests causal models.

A fundamental issue is what your theory predicted the likely factorial structure to be, and whether a 1, 2, 3 ... factor model is theoretically supportable. Could the two-factor solution be 'telling' you something interesting/new? My feeling is that single factor models are (almost) always likely to be over-simple because the world is too complex for multiple outcomes (e.g. collected from Likert questionnaire items) to have a single cause. However, even in a two-factor model it is always open to a researcher to argue that one of the two factors is the predominant cause of observed behaviour.