Satorra-Bentler chi-square difference test

[Stephanie Fleischer]

Dear all,
I want to check the longitudinal measurement invariance of my dependent variable. Therefore, I am doing a longitudinal confirmatory analysis
use the Satorra-Bentler adjustments because the items – I use – are not normally distributed. I run three models (configural, metric and scalar). Now, I want to run the Satorra-Bentler chi-square difference test to compare the models.
My question is: Is it possible to run the Satorra-Bentler chi-square difference test with STATA 19? I only find ways to do this test with MPlus or R.
Thanks for your help.

[Nick Cox]

https://www.stata.com/features/overv...torra-bentler/ may help.

[Erik Ruzek]

Unfortunately, there is not an easy way to do this using lrtest and as far as I can tell, no one has written a package to do it. You will either need to write your own program to do it (see here for the details of the calculation) or utilize this handy Excel document created by Natasha Bowen, who graciously posted it to the internet with documentation.

[Erik Ruzek]

With some assistance from Google's Gemni, I was able to come up with the following, which also reports the difference in the CFI, SRMR, and RMSEA. These differences are often reported when doing measurement invariance testing, in addition to the chi-square difference test. I do not claim this is perfect, but everything looks right to me. There are other expert Stata programmers on here who might be able to suggest improvements or point out problems.

cap program drop sb_diff_all
program sb_diff_all
    version 14
    args m0 m1  // m0 = restricted (null), m1 = unrestricted (alt)

    * 1. Extract values for Model 0 (Restricted)
    quietly estimates restore `m0'
    quietly estat gof, stats(all)
    tempname t0 sb0 df0 c0 cfi0 rmsea0 srmr0
    scalar `t0'     = e(chi2_ms)
    scalar `sb0'    = e(chi2sb_ms)
    scalar `df0'    = e(df_ms)
    scalar `c0'     = `t0' / `sb0'
    scalar `cfi0'   = r(cfi)
    scalar `rmsea0' = r(rmsea)
    scalar `srmr0'  = r(srmr)

    * 2. Extract values for Model 1 (Unrestricted)
    quietly estimates restore `m1'
    quietly estat gof, stats(all)
    tempname t1 sb1 df1 c1 cfi1 rmsea1 srmr1
    scalar `t1'     = e(chi2_ms)
    scalar `sb1'    = e(chi2sb_ms)
    scalar `df1'    = e(df_ms)
    scalar `c1'     = `t1' / `sb1'
    scalar `cfi1'   = r(cfi)
    scalar `rmsea1' = r(rmsea)
    scalar `srmr1'  = r(srmr)

    * 3. Calculate Satorra-Bentler Scaled Difference
    tempname df_diff cd t_diff p_val dcfi drmsea dsrmr
    scalar `df_diff' = `df0' - `df1'
    scalar `cd'      = (`df0' * `c0' - `df1' * `c1') / `df_diff'
    scalar `t_diff'  = (`t0' - `t1') / `cd'
    scalar `p_val'   = chi2tail(`df_diff', `t_diff')
    
    * Calculate Absolute Differences
    scalar `dcfi'    = `cfi1' - `cfi0'
    scalar `drmsea'  = `rmsea1' - `rmsea0'
    scalar `dsrmr'   = `srmr1' - `srmr0'

    * 4. Display Results
    di _n as text "Satorra-Bentler Scaled Chi-Square Difference Test"
    di as text "{hline 60}"
    di as text "Restricted Model (M0):   " as result "`m0'"
    di as text "Unrestricted Model (M1): " as result "`m1'"
    di as text "{hline 60}"
    di as text "Difference Chi-square:   " as result %8.3f `t_diff'
    di as text "Degrees of Freedom:      " as result `df_diff'
    di as text "p-value:                 " as result %8.4f `p_val'
    di as text "{hline 60}"
    di as text "Fit Index Change (M1 - M0):"
    di as text "Delta CFI:               " as result %8.4f `dcfi'
    di as text "Delta RMSEA:             " as result %8.4f `drmsea'
    di as text "Delta SRMR:              " as result %8.4f `dsrmr'
    di as text "{hline 60}"
    di as text "Note: SB-adjusted indices are used where available."
end

Now try it out with some data. Note, that you would need to specify separate models for each of the types of invariance models. Here, I'm just adding one parameter between m0 and m1, and not looking at invariance, but the general idea is similar.

* 1. Load data
use https://www.stata-press.com/data/r19/sem_1fmm.dta
* 2. Run program
sem (F1 -> x1-x4), vce(sbentler)
est sto m0
sem (F1 -> x1 x2 x3 x4), cov(e.x2*e.x3) vce(sbentler)
est sto m1
sb_diff_all m0 m1

And the output:

Satorra-Bentler Scaled Chi-Square Difference Test
------------------------------------------------------------
Restricted Model (M0):   m0
Unrestricted Model (M1): m1
------------------------------------------------------------
Difference Chi-square:      1.448
Degrees of Freedom:      1
p-value:                   0.2289
------------------------------------------------------------
Fit Index Change (M1 - M0):
Delta CFI:                 0.0000
Delta RMSEA:               0.0000
Delta SRMR:               -0.0052
------------------------------------------------------------
Note: SB-adjusted indices are used where available.

[Stephanie Fleischer]

Unfortunately, there is not an easy way to do this using lrtest and as far as I can tell, no one has written a package to do it. You will either need to write your own program to do it (see here for the details of the calculation) or utilize this handy Excel document created by Natasha Bowen, who graciously posted it to the internet with documentation.

I found similar explanations on how to calculate the chi-sqaure differnce test. Unfortunately, I was't sure how to get the scaling factor out of the STATA output. But I guess found the calculation of the scaling factor in your second post. Thank you!
The link to the excel sheet ist very helpful.

[Stephanie Fleischer]

With some assistance from Google's Gemni, I was able to come up with the following, which also reports the difference in the CFI, SRMR, and RMSEA. These differences are often reported when doing measurement invariance testing, in addition to the chi-square difference test. I do not claim this is perfect, but everything looks right to me. There are other expert Stata programmers on here who might be able to suggest improvements or point out problems.

cap program drop sb_diff_all
program sb_diff_all
version 14
args m0 m1 // m0 = restricted (null), m1 = unrestricted (alt)

* 1. Extract values for Model 0 (Restricted)
quietly estimates restore `m0'
quietly estat gof, stats(all)
tempname t0 sb0 df0 c0 cfi0 rmsea0 srmr0
scalar `t0' = e(chi2_ms)
scalar `sb0' = e(chi2sb_ms)
scalar `df0' = e(df_ms)
scalar `c0' = `t0' / `sb0'
scalar `cfi0' = r(cfi)
scalar `rmsea0' = r(rmsea)
scalar `srmr0' = r(srmr)

* 2. Extract values for Model 1 (Unrestricted)
quietly estimates restore `m1'
quietly estat gof, stats(all)
tempname t1 sb1 df1 c1 cfi1 rmsea1 srmr1
scalar `t1' = e(chi2_ms)
scalar `sb1' = e(chi2sb_ms)
scalar `df1' = e(df_ms)
scalar `c1' = `t1' / `sb1'
scalar `cfi1' = r(cfi)
scalar `rmsea1' = r(rmsea)
scalar `srmr1' = r(srmr)

* 3. Calculate Satorra-Bentler Scaled Difference
tempname df_diff cd t_diff p_val dcfi drmsea dsrmr
scalar `df_diff' = `df0' - `df1'
scalar `cd' = (`df0' * `c0' - `df1' * `c1') / `df_diff'
scalar `t_diff' = (`t0' - `t1') / `cd'
scalar `p_val' = chi2tail(`df_diff', `t_diff')

* Calculate Absolute Differences
scalar `dcfi' = `cfi1' - `cfi0'
scalar `drmsea' = `rmsea1' - `rmsea0'
scalar `dsrmr' = `srmr1' - `srmr0'

* 4. Display Results
di _n as text "Satorra-Bentler Scaled Chi-Square Difference Test"
di as text "{hline 60}"
di as text "Restricted Model (M0): " as result "`m0'"
di as text "Unrestricted Model (M1): " as result "`m1'"
di as text "{hline 60}"
di as text "Difference Chi-square: " as result %8.3f `t_diff'
di as text "Degrees of Freedom: " as result `df_diff'
di as text "p-value: " as result %8.4f `p_val'
di as text "{hline 60}"
di as text "Fit Index Change (M1 - M0):"
di as text "Delta CFI: " as result %8.4f `dcfi'
di as text "Delta RMSEA: " as result %8.4f `drmsea'
di as text "Delta SRMR: " as result %8.4f `dsrmr'
di as text "{hline 60}"
di as text "Note: SB-adjusted indices are used where available."
end

Now try it out with some data. Note, that you would need to specify separate models for each of the types of invariance models. Here, I'm just adding one parameter between m0 and m1, and not looking at invariance, but the general idea is similar.

* 1. Load data
use https://www.stata-press.com/data/r19/sem_1fmm.dta
* 2. Run program
sem (F1 -> x1-x4), vce(sbentler)
est sto m0
sem (F1 -> x1 x2 x3 x4), cov(e.x2*e.x3) vce(sbentler)
est sto m1
sb_diff_all m0 m1

And the output:

Satorra-Bentler Scaled Chi-Square Difference Test
------------------------------------------------------------
Restricted Model (M0): m0
Unrestricted Model (M1): m1
------------------------------------------------------------
Difference Chi-square: 1.448
Degrees of Freedom: 1
p-value: 0.2289
------------------------------------------------------------
Fit Index Change (M1 - M0):
Delta CFI: 0.0000
Delta RMSEA: 0.0000
Delta SRMR: -0.0052
------------------------------------------------------------
Note: SB-adjusted indices are used where available.

Thank you for the code. I will try it with my data.