IRT graded Response Model Query

[Paul Bowen]

Dear All

My apologies for contacting you in the forum space, but I really would appreciate your guidance.

A colleague and I are evaluating a 3-item scale for self-perceived sleep problems of construction workers. There are 5 response options: “none of the time” to “all of the time” - scored 1 to 5. A GR IRT model is being used. One of the items is problematic.

My colleague got the GR model to work as follows: "I was able to get it to work for the GR model once I imposed a tight prior on the a for Sleep_ProblemR (i.e., normal with mean 0, SD = 1). I will dig deeper into this issue and a rationale for this prior choice.”

The problem is, my colleague uses IRTPRO and I use STATA.

My question then is: Using STATA, how do I impose a tight prior on ‘a’ for the item Sleep_ProblemR (i.e., normal, with mean = 0 and SD = 1)?

I know precious little about STATA programming and syntax, preferring to use the menus. Please assume that I am a complete novice, because I am!!

Thank you.

Best wishes
Paul Bowen

University of Cape Town
Cape Town, South Africa

[Weiwen Ng]

You might want to clarify what a is. I'm assuming it's the alpha parameter, which is the item discrimination parameter in the parameterization that Stata used.

When you say one of the items is problematic, do you mean that the model is not converging in both of your programs? I am not a Bayesian, but a standard normal prior doesn't sound like a reasonable choice for a prior on a discrimination parameter to me - most discrimination parameters I've seen in the health services research lit are over 1, and often well over 1. I have to wonder if there are more fundamental problems here than just model convergence? It is only three items, so I'd expect things to converge. FYI, almost all of Stata's commands are within a frequentist paradigm. If a was actually the discrimination parameter, then there is no way that I know to get Stata's native IRT command to assume a prior. All the IRT commands use the standard normal distribution as a prior for the subject ability level, but that's it.

Stata 15 does allow you to fit Bayesian IRT models. I'm reading the documentation, and the syntax is rather complex. Perhaps assuming that the log of the discrimination parameter for that item is standard normal would be helpful, but it does also strike me that then, the entire analysis should be done in Bayesian format. But I'm afraid that doesn't help you with setting that command up. Sorry, but that's well beyond my ability right now. Perhaps someone else can be more helpful.

[Paul Bowen]

Thank you Weiwen. Yes, the 'a' is the discrimination parameter. The Stata program does not converge at all. The IRTPRO GR model does converge, but the problematic item has an 'a' value of 25.X, whilst the other 2 items have 'a' values of about 1.X. The ICC and IIF curves for the problematic first item look very dubious.

When my colleague, using IRTPRO, imposed the tight prior on the 'a' for the problematic Sleep_ProblemR item, (i.e., normal with mean 0, SD = 1), the results were good and the curves looked perfectly fine.

The problem lies in getting Stata to replicate that outcome.

Thanks for your inputs,
Paul

[Weiwen Ng]

Paul, in that case, I can't think of anything that would really help you within Stata's IRT framework, but I am no IRT expert, and perhaps someone else will chime in.

I wonder: is it standard practice in IRT models to assess whether the items are sufficiently unidimensional? It really seems like the item in question is problematic somehow. With respect, if I were a reviewer, and I knew that the model only fit if you imposed a pretty specific prior on the parameter, I would be a bit skeptical of the overall model. Also, as stated above, I'm not certain that the prior is proper - forgive me if the term isn't correct, but I mean that we know that alpha parameters have to be positive, and if your colleague imposed a standard normal prior for alpha, that prior exists on the entire real line.

I don't believe that exploratory factor analysis will work in your case, as you have only 3 categorical items. However, some IRT papers I've seen use Loevinger's H coefficients (for each item and for the overall model) to assess how well the items conform to a Guttman scale. If you want to try this, there is a user-written Stata routine for this by Jean-Benoit Hardouin on SSC; type

ssc install loevh

[Weiwen Ng]

I may have spoken too hastily. I don't routinely fit IRT models, but I know that in general, some likelihood functions can be vulnerable to local maxima. If the graded response model likelihood is one case, then it might be possible to fix this by specifying a different starting location.

I know you asked to use the menus, but I'm not sure this is possible from the menu system, and I'm only familiar with the command line interface. But, here is some sample code.

webuse charity /*This is a sample dataset hosted by StataCorp*/
irt grm ta1-ta5 /*Fit a graded response IRT model with the variables ta1 to ta5*/
irt grm ta1-ta5, iterate(2) /*Re-fit the same model, but perform only 2 iterations when maximizing the likelihood*/
matrix b = e(b) /*This extracts the current matrix of parameter estimates.*/
matrix list b /*This displays the current matrix. Pretend that the alpha parameter for question ta1 was the problematic one. You refer to that parameter by b[1,1].*/
matrix b[1,1] = 1 /*Manually change the parameter estimate to 1; this probably isn't a crazy starting value, given what we know about the properties of discrimination parameters*/
irt grm ta1-ta5, from(b) /*Re-fit the GRM, starting with the parameter estimates from what you just produced. Compare to the original results from the model that converged, and they're very similar.*/

In real life, you can find the iteration number where the log-likelihood stops increasing, then tell the program to stop iterating then. For example:

irt grm q1-q3
/*Pretend the program hit 100 iterations, then the log likelihood stopped increasing and the program kept iterating forever. Press control-break.*/
irt grm q1-q3, iterate(100)
matrix b = e(b)
matrix b[1,1] = 1
irt grm q1-q3, from(b)

This may work. Not guaranteed. You could use the final parameter estimate from your colleague's work as a starting value if you prefer. This is definitely not imposing a Bayesian prior on the data. You're merely telling the program you want to start maximizing the likelihood from a different starting spot. If it converges, the solution is (I believe) accurate and acceptable. Or your colleague can try this approach within IRTPRO if he or she likes. If this approach doesn't make sense, then I'd approach someone who is more familiar with IRT models. As usual, if I'm completely off base, I hope to hear from others!

[Paul Bowen]

Thank you Weiwen. I will try this out; Much appreciated. Paul

[Paul Bowen]

Hi Weiwen

Wonderful; it worked with the following command. The parameter values were not absolutely identical to those yielded by IRTPRO, but were very close. Thanks. Paul
irt grm ta1-ta5, iterate(2)

[Weiwen Ng]

Hi Weiwen

Wonderful; it worked with the following command. The parameter values were not absolutely identical to those yielded by IRTPRO, but were very close. Thanks. Paul
irt grm ta1-ta5, iterate(2)

I am glad to hear that this resolved the issue!

[Paul Bowen]

Hi Weiwen
Why is the magical number 2 iterations? I tried other numbers of iterations e.g., 1, 3, 4, and their parameter values are not close to the IRTPRO outcomes. How did you know that up-front?

[Weiwen Ng]

Hi Weiwen
Why is the magical number 2 iterations? I tried other numbers of iterations e.g., 1, 3, 4, and their parameter values are not close to the IRTPRO outcomes. How did you know that up-front?

Maybe I should have clarified things!

Stata maintains a list of stock datasets. The set of code I gave you was written to work with that particular dataset. In fact, it's the stock dataset for the IRT GRM command example in the manual.

In the sequence of code I gave you, I was trying to get Stata to terminate its maximization routine before convergence. I chose 2 iterations because I knew that the model converged in something like 5 iterations. If you tell Stata to do 2 iterations and stop, then it will still produce a matrix of parameters (that's stored in e(b), and treated as a matrix). Then, you take the problematic parameter, and you give it some value you think is close to reality. You have to take the matrix of estimated parameters and assign it a proper name before you can manipulate any of the cells.

I arbitrarily chose a value of 1, since I typically hear that we want discrimination parameters at least 1, and we know that less than 0 is impossible. You could input any reasonable value; you could try inputting the final parameter estimate from your colleague's analysis if you like. This is one way I know about to get around problems with convergence. This could help if the likelihood function had some local maxima that it was converging on instead of the global maximum.

If absolutely none of the above made sense, then it would be best to consult a real statistician in person - I'm an applied statistician at best, and we can only do so much on a forum. That said, if you are generally familiar with programming from a command line, then if you install a user-written command called dataex, you can get code that enables you to post an example of your code here. For example, if your variables have the names q1, q2, and q3, then type:

ssc install dataex
dataex q1 q2 q3, count(50)

That would produce an extract of 50 observations with just the variables q1-q3, and you can post that on the forum using the code delimiter (or hit the # button on the formatting toolbar on your screen).

[Giovanni Piumatti]

Hello,

Sorry if my questions may be out of context in this thread: is there a way to calculate differential item functioning using the grade response model in stata?

thank you

Giovanni Piumatti
University of Geneva

[Weiwen Ng]

Hello,

Sorry if my questions may be out of context in this thread: is there a way to calculate differential item functioning using the grade response model in stata?

thank you

Giovanni Piumatti
University of Geneva

Giovanni, Stata does allow testing for DIF, as outlined in the manual. Type

help dif

from the command line. If you have any specific questions on the DIF commands, please start a separate thread.

[Paul Bowen]

Hello
In IRTPRO 4.1 one can specify the method of estimation e.g., Bock-Aitkin, Adaptive Quadrature, MH-MR, or MC-MC; as well as unidimensionality or multidimensionality. One can also set parameter values to be equal, free, or fixed. All of this is menu driven.

How do I do this in Stata, either by menus or by commands?

I would be grateful for your guidance. Specific instructions would be appreciated. Thank you.
Paul Bowen

[Weiwen Ng]

Hello
In IRTPRO 4.1 one can specify the method of estimation e.g., Bock-Aitkin, Adaptive Quadrature, MH-MR, or MC-MC; as well as unidimensionality or multidimensionality. One can also set parameter values to be equal, free, or fixed. All of this is menu driven.

How do I do this in Stata, either by menus or by commands?

I would be grateful for your guidance. Specific instructions would be appreciated. Thank you.
Paul Bowen

In Stata, the default likelihood maximization algorithm is mean-variance adaptive Gauss-Hermite quadrature. 3 other maximization routines are available, at least one of which is another form of adaptive quadrature. If you want to use Bayesian estimation (i.e. the Markov Chain Monte Carol or MCMC method you referred to), you would have to learn to use the Bayesian syntax, but this can definitely be done (not that I have done it). See the section called integration under the IRT manual: https://www.stata.com/manuals/irt.pdf. I'm not familiar with Bock-Aitkin or MH-MR, so I can't comment, but it doesn't look like Stata incorporates these. I have no idea how to do this from the menus, but I imagine it can be done.

IRT models as fit by our IRT suite are inherently unidimensional. If you want to fit an multidimensional IRT model, you will have to specify that in -gsem-. I know that sort of model exists, but I have not fit one successfully. I see that a user written command for a certain type of multidimensional rating scale model exists, written by Jean-Benoit Hardouin. If that's not what you meant, please clarify.

[Weiwen Ng]

Hello
In IRTPRO 4.1 one can specify the method of estimation e.g., Bock-Aitkin, Adaptive Quadrature, MH-MR, or MC-MC; as well as unidimensionality or multidimensionality. One can also set parameter values to be equal, free, or fixed. All of this is menu driven.

I forgot to answer the part about constraining parameters. Yes, you can constrain some parameters to be equal, or some to be fixed at some value, but you have to use the -gsem- command. See SEM example 8. Again, I suppose this is possible in the menu, but I have no idea how to do it.

As I said, you'll have to use our -gsem- command; this presentation gave an example of how to fit an IRT model in -gsem-.