Item fit (outfit/infit) statistics using -irt-?

[Brandon Olszewski]

I recently upgraded to Stata 14 in part because of the new IRT capabilities. In the past, I've used Winsteps for item analysis, largely around assessing item fit for purposes of assessment development. I've used fit statistics in ways similar to those described in Smith's (2000) paper in the Journal of Applied Measurement, depending on infit and outfit indices to assess item fit to improve overall assessment reliability.

I love the options in Stata 14 for fitting different (pcm, grm) models, but as far as I can tell, Stata 14 does not provide fit statistics for items. So, two questions:

1. Am I wrong here? Can the IRT family of commands provide fit (infit/outfit) statistics? I searched the forum using "item fit IRT" and did not find what I needed, although one post (http://www.statalist.org/forums/foru...tent-variables) did allude to testing goodness of fit using an SEM model...which is not what I want to do.

2. If the IRT family cannot supply fit statistics for items, is there a way to assess item fit given the existing commands currently available?

Thanks for any help.

[wbuchanan]

You would probably need to compute infit and outfit statistics using the returned matrices. I think there was also something that was part of the FreeIRT project that produced those estimates but can't remember the name of the package on SSC that has them. Also, it'd be helpful to use full citations in case others who are less familiar with the field have an interest in the area could provide help from viewing the literature.

[Brandon Olszewski]

Dang. I figured as much.

If anyone happens to know a novice-friendly source for how one might go about calculating infit/outfit estimates from the statistics natively available in Stata, I would certainly love the help. Scripting this would be a valuable contribution to the community - a lofty goal for my first user-written package. I would really love to be able to do this in Stata. While I have found Winsteps to be fairly easy to use, it only provides estimates of discrimination, and can't account for pseudo-guessing. I'm also not sure if it does graded response models, which would be helpful since I work with a lot of Likert-type data.

Apologies on the vague citation, and good point. Smith (2000) article "Understanding Rasch Measurement: Fit Analysis in Latent Trait Measurement Models" was helpful to me (Journal of Applied Measurement, vol 1 no 2). http://jampress.org/pubs.htm

I also looked into the FreeIRT project (http://www.freeirt.org/). Looks like it's closed. It referred me to a different site that included some Stata packages, none of which compute fit statistics by item, unfortunately. But thank you for citing that source, and for the reply.

[wbuchanan]

Brandon Olszewski I just finished coding up an early program that uses that Java API in Stata to interface with the back end of jMetrik.

net inst raschjmle, from("http://www.paces-consulting.org/stata")

The program estimates both item and person level infit, outfit, standardized infit, and standardized outfit statistics in addition to estimating the theta and beta parameters and corresponding standard errors. As long as your data are dichotomously keyed it should work for you.

[Brandon Olszewski]

wbuchanan
I just saw this. Wow - thanks! My apologies for replying so late.
I will definitely try it out!

[wbuchanan]

Brandon Olszewski no worries. There is probably a way of computing things from the returned matrices, but it likely would involve a fair amount of manipulation given the way the IRT models are parameterized in Stata.

[Nathan Roberson]

wbuchanan
Like Brandon, I am also trying to get fit statistics in STATA.

I downloaded your package in STATA, but I keep getting the following error:
java.lang.NoClassDefFoundError: org/paces/Stata/IRT
Caused by: java.lang.ClassNotFoundException: org.paces.Stata.IRT
at java.net.URLClassLoader$1.run(URLClassLoader.java: 372)
at java.net.URLClassLoader$1.run(URLClassLoader.java: 361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.j ava:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:4 24)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:3 57)

Do you know why this might be?

[wbuchanan]

Hi Nathan,

Which version of the program are you using? And which version of Java do you have installed on your system? If you're able to get back soon enough I may be able to have a fix in place by the morning. Also, it is worth mentioning that this will only provide the infit/outfit statistics for the Joint MLE with binary items. I've not yet gotten around to implementing the PCM versions of the underlying Java interface. I'll look into things a bit in the morning in either case.

[Nathan Roberson]

Thanks for the reponse wbuchanan,

I have Java V. 8.73 installed and I'm using STATA 14.1. Does it matter that I am trying to run this on a Mac?

And yes, my data is binary. Although if you implement a PCM version I would definitely use that too!

[wbuchanan]

I think it may have been an issue with the Java binary that was on the server previously. So, the best thing to do at this point is:

ado, find(raschjmle)
ado uninstall [#]
net inst raschjmle, from("http://wbuchanan.github.io/StatajMetrik/")

If you install it and can't locate the file "raschjmle-jar-with-dependencies.jar" on your ADO path let me know. Additionally, there is now a generic project page as well as a web-friendly version of the helpfile with an example posted now. I'd say this would be the best location moving forward to check for updates since it requires a significant amount less effort to maintain/update and also is integrated with the version control system I use.

Generally, Java programs/plugins should operate the same regardless of the OS since they run in the Java Virtual Machine; you can think of it almost like a computer within the computer where it handles the communication with your hardware. There are a few limited examples where this isn't the case (see Stata File System as an example), but in these cases there are methods available for the different platforms. I haven't dug into the documentation for the psychometrics library in a bit, but I know from conversations with Patrick Meyer that there are methods for using the JML estimator with polytomous items (its less clear whether the infit/outfit statistics generalize, but it would seem strange that the primary exposition of those statistics would be in a book on rating scales and not generalize to polytomous items).

[wbuchanan]

As a brief update, the version available from the project page listed above can now be used in Stata 13 or later.

[Ajay Gupta]

"wbuchanan" Sir, I need your help urgently. Kindly help me in rasch anlaysis. I am trying to do it for half month.

[wbuchanan]

Ajay, you need to provide significantly more information than this in order for anyone to know what specifically you need help with.

[Cristian Vaccari]

@wbuchanan many thanks for putting together this package. It generate lots of statistics. Would you be able to briefly elaborate on the statistics that are most helpful to evaluate goodness of fit?

I have fitted two different 1pl IRT models on binary variables and I get the following data from your package.

First model:

SCALE QUALITY STATISTICS
==================================================
Statistic Items Persons
--------------------------------------------------
Observed Variance 0.0011 0.0000
Observed Std. Dev. 0.0335 0.0000
Mean Square Error 0.0079 2.0006
Root MSE 0.0888 1.4144
Adjusted Variance 0.0000 0.0000
Adjusted Std. Dev. 0.0000 0.0000
Separation Index 0.0000 0.0000
Number of Strata 0.3333 0.3333
Reliability 0.0000 NaN
==================================================


Second model:


SCALE QUALITY STATISTICS
==================================================
Statistic Items Persons
--------------------------------------------------
Observed Variance 0.2426 0.0000
Observed Std. Dev. 0.4925 0.0000
Mean Square Error 0.0196 2.1238
Root MSE 0.1399 1.4573
Adjusted Variance 0.2230 0.0000
Adjusted Std. Dev. 0.4722 0.0000
Separation Index 3.3754 0.0000
Number of Strata 4.8339 0.3333
Reliability 0.9193 NaN
==================================================

Many thanks in advance!

[wbuchanan]

Cristian Vaccari
I'd suggest checking out the documentation for jMetrik to get the best explanation of the results since these are all metrics defined in that program's internals.