3pl IRT with Different Parameter Estimates for Groups

[Katelyn Stauffer]

Hi All,

I'm working on a project in which I need to construct a knowledge scale. I have about 15 items (respondents are a 1 if they answer correctly; 0 if they are incorrect or answer "don't know"). I want to use a 3pl IRT model to estimate the underlying knowledge scale; the problem is that men are significantly more likely to guess on a question they aren't sure of, while women are more likely to answer don't know. So, I end up with a gap in knowledge that is partially a function of men's propensity to guess. Is there a way to estimate this model with different parameters (in this case a different guessing parameter) for the different genders? Many thanks in advance!

[wbuchanan]

First step would be to test which model fits the data best. It may be that adding a third parameter makes the model fit the data worse. After that, you would use DIF to determine whether or not the items function differently. There would be no way to determine how to handle cases when they are coded identically even though the values have a potentially different meaning across the groups. Another option would be to fit a mixed Rasch model and/or another analogue. I'm not sure why you would code "don't know" responses the same as incorrect since in one case the data are missing/invalid and the other is a valid response? Not sure whether the IRT commands use a full information estimator, but as long as they do, you could probably get cleaner estimates by avoiding the scoring/keying issue with the responses.

[wbuchanan]

Katelyn Stauffer it may be possible to do something like this using different priors in a Bayesian framework as well. Nikolay Balov (StataCorp) and Yulia Marchenko (StataCorp) put together an extremely helpful blog for using the bayesmh command to fit IRT models in Stata. It may be possible to specify a prior for the c parameter with greater location parameter for men vs women, but it isn't anything that I would know how to specify and I'm not even sure that would be sufficient given the coding conundrum.