Calculating ordinal alpha using polychoric factor analysis - include file question

[Molly King]

I am trying to develop code to calculate ordinal alpha as an alternative to Cronbach's alpha when developing scales or calculating model reliability based on ordinal variables. Through use of previous Statalist postings (Ordinal Alpha, calculate alpha after polychoric factor analysis), I have written an include file that calculates ordinal alphas based on Joseph Coveney's suggested code. However, my code produces ordinal alpha results which do not vary with the values of my input variables (local `variable_set' in the include file). The only thing that seems to cause the ordinal alpha results to vary is the number of variables in my `variable_set', not their values.

Below is the relevant code from my main .do file to run the include file, my include file, and sample data. I am hoping for help with understanding what I am doing wrong in my code such that I am not getting any variance (based on the original values of my Likert scales) in the final ordinal alpha value produced. Thank you in advance.

I am using Stata v13, though I have access to v15 if that would solve the problem.

Here's the main code to set locals and reference the include file:

rename (soccer_news soccer_talks soccer_books soccer_learn soccer_know soccer_und_e) latent#, addnumber(1)
local variable_set "latent1 latent2 latent3 latent4 latent5 latent6"
include ordinalAlpha.doi

*Code to run ordinal alpha results for only 5 variables:
drop latent6    
local variable_set "latent1 latent2 latent3 latent4 latent5"
include ordinalAlpha.doi

Here is the include file:

*! ordinalAlpha.doi
*! Calculate ordinal alpha after polychoric factor analysis
*! (Appropriate factor analysis for likert scale)
*! Based on code written by Joseph Coveney:
*! https://www.stata.com/statalist/archive/2012-02/msg00696.html

/*
Relevant literature:
Bruno D. Zumbo, Anne M. Gadermann and Cornelia Zeisser. 2007. 
    Ordinal Versions of Coefficients Alpha and Theta for Likert Rating Scales. 
    _Journal of Modern Applied Statistical Methods_ 6(1): 21--9.
AND
Anne M. Gadermann, Martin Guhn & Bruno D. Zumbo. 2012
    Estimating ordinal reliability for Likert-type and ordinal item response
    data: A conceptual, empirical, and practical guide.
    _Practical Assessment, Research & Evaluation_ 17(3).
*/

***--------------------------***
// # PROGRAM INPUT
***--------------------------***

/*
To run this include file, you need 2 pieces of input in a .do file:
    1) A local variable with your variables of interest renamed in this format:
            local variable_set "latent1 latent2 latent3..."
    2) A pointer to this include file, something like:
        include ordinalAlpha.doi
*/

***--------------------------***
// # PROGRAM SETUP
***--------------------------***


version 13
set more off
set seed 1
tempname C

*count how many variables are in variable_set and set that as a local
*(https://www.statalist.org/forums/forum/general-stata-discussion/general/1249342-counting-the-number-of-variables-with-same-stub)
ds `variable_set'
local num :  word count `r(varlist)'
display `num'
display wordcount("`r(varlist)'")

***--------------------------***
// # POLYCHORIC MATRIX
***--------------------------***


matrix define `C' = I(`num') * 0.45 + J(`num', `num', 0.55)
drawnorm `variable_set', corr(`C') double n(`sample_size') clear

local min = 1 /(`num'+1)
local increment = 1 / (`num'+1)
local max = 1-`increment'

forvalues i = 1/`num' {
    generate double u`i' = normal(latent`i')
    generate byte manifest`i' = 0
    quietly forvalues cut = `min'(`increment')`max' {
        replace manifest`i' = manifest`i' + 1 if u`i' > `cut'
    }
}

polychoric manifest*
matrix define `C' = r(R)
factormat `C', n(`sample_size') factors(1)

display in red "Which items to include in factor model? Commonly used convention is to only consider items with factor loadings greater than .40 (Ford, MacCallum, & Tait 1986)"
display in red "In other words, Factor1  > .40"

tempname L Psi
matrix define `L' = e(L)
matrix define `Psi' = e(Psi)

local p = rowsof(`L')

tempname f f2 u2
scalar define `f' = 0
scalar define `f2' = 0
scalar define `u2' = 0

forvalues i = 1/`p' {
    scalar define `f' = `f' + `L'[`i', 1]
    scalar define `f2' = `f2' + `L'[`i', 1] * `L'[`i', 1]
    scalar define `u2' = `u2' + `Psi'[1, `i']
}

scalar define `f' = `f' / `p'
scalar define `f2' = `f2' / `p'
scalar define `u2' = `u2' / `p'

tempname pf2
scalar define `pf2' = `p' * `f' * `f'
scalar define alpha = `p' / (`p' - 1) * ///
    (`pf2' - `f2') / (`pf2' + `u2')
    
***--------------------------***
// # ORDINAL ALPHA RESULTS
***--------------------------***

display in red "Ordinal alpha = " as result %06.4f alpha
display in red "The psychometric literature (e.g., Nunnally, 1978) recommends that alpha for a scale should not be smaller than .70 for research purposes"
*alpha latent*
alpha manifest*, std

exit

Here is some example data:

* Example generated by -dataex-. To install: ssc install dataex
clear
input byte(soccer_talks soccer_learn soccer_books soccer_know soccer_und_e laser_news_w laser_talks_w laser_books_w laser_learn_w laser_know_w laser_und_t)
3 3 4 3 2 7 7 7 7 7 4
2 2 3 5 3 5 6 5 6 7 4
3 4 3 2 1 6 6 6 6 6 7
1 1 1 3 1 2 2 1 3 1 7
1 2 1 1 3 6 6 6 7 7 6
2 2 3 2 2 7 7 7 7 7 7
1 3 5 2 2 7 6 6 6 6 6
1 2 4 1 1 7 7 7 7 7 7
1 1 1 1 2 6 5 6 7 6 7
1 1 1 1 3 6 6 6 6 6 7
3 3 3 3 2 4 4 4 4 4 7
2 2 4 1 4 4 4 4 4 6 6
2 2 3 2 3 5 6 5 6 6 6
3 3 3 3 6 5 4 6 7 5 5
4 4 4 4 5 5 6 6 5 4 4
end

[daniel klein]

Molly

First, the code that you include does not seem to use the values of the variables that you supply but only the number of supplied variables. It then seems to create an example dataset; you have already noticed that. Now, I would not even try to fix this. If you want to use the suggested code, you really should turn this into a program (or ado-file). Here is a quick sketch

capture program drop ordinalalpha
program ordinalalpha
    version 13
    
    syntax varlist(numeric)
    
    tempname C
    polychoric `varlist'
    local N = r(N)
    matrix `C' = r(R)
    factormat `C' , n(`N') factors(1)
    
    tempname L Psi
    matrix define `L' = e(L)
    matrix define `Psi' = e(Psi)

    local p = rowsof(`L')

    tempname f f2 u2 alpha
    scalar define `f' = 0
    scalar define `f2' = 0
    scalar define `u2' = 0

    forvalues i = 1/`p' {
        scalar define `f' = `f' + `L'[`i', 1]
        scalar define `f2' = `f2' + `L'[`i', 1] * `L'[`i', 1]
        scalar define `u2' = `u2' + `Psi'[1, `i']
    }

    scalar define `f' = `f' / `p'
    scalar define `f2' = `f2' / `p'
    scalar define `u2' = `u2' / `p'

    tempname pf2
    scalar define `pf2'   = `p' * `f' * `f'
    scalar define `alpha' = `p' / (`p' - 1) * (`pf2' - `f2') / (`pf2' + `u2')
    
    display as txt "Ordinal alpha = " as result %06.4f `alpha'
end

Note that polychoric (from http://staskolenikov.net/stata) is a community contributed program that you need to install. Anyway, include the program at the beginning of your do-file, then call it as

ordinalalpha varlist

However, I do not think that the code calculates the (equivalent of Cronbach's) alpha coefficient. If I remember correctly, alpha assumes tau-equivalence, meaning that it restricts the factor loadings to be equal. In fact, the calculation of alpha does not even require factor loadings. Yet, the code that Joseph provided is based on factor loadings; this is probably superior to alpha, since tau-equivalence seldom holds in real-world-data. You may want to read into the cited literature to get a better understanding of what it is you are calculating here so you can report it more detailed and reduce the risk of misunderstandings.

Last, as has already been pointed out in the post on (old) Statalist that you cite, you might be better off, estimating an appropriate GSEM model and estimate reliability from that model. A more crude and completely different but simple solution would be treating the Items, which seem to have a range of 1-7, based on what you show, as interval scaled and just use alpha.

Best
Daniel

[Molly King]

Daniel,
Thank you very much for taking the time to answer my less-than-straightforward question. I really appreciate it.
Molly