Problem with IRT 2PL Convergence

[surendra kumar labanya]

Dear Sir/Madam,
I have gone through the previous two post on IRT 2PL methods and am facing the same problem related to IRT 2PL convergence. Therefore, I request you to provide the solution for the above mentioned problem. Since the solution given in the previous post does not cover the issue related to IRT 2Pl method, it will be a great help to us if you provide us with the solution.

[Weiwen Ng]

Surendra,

I assume you're referring to this post and this post.

This question is very general, and I assume that this is why there were few responses. Convergence issues are tricky, and I am not familiar with the specific issues that would prevent convergence in a 2PL model.

In the first post linked above, the poster dropped questions that had no variation, i.e. everyone got them wrong or everyone got them correct. I haven't tested this scenario, but if everyone gets a question wrong, the MLE of the difficulty parameter is infinity, and that might prevent convergence. I assume you've checked this. It would be nice if things were as simple as that. Unfortunately, that poster still (may have) had convergence trouble. Another more complex issue was outlined by wbuchanan in post #6 on the second thread: if you have a small sample and your set of items isn't unidimensional, there might be a problem there as well.

Stata 17 has a maximum of 300 iterations. I think Stata 16 may have that set at something over 10,000, which is way more than you need. If you have Stata 17, you could just let the program run to 300 iterations and then examine the output. Which questions have missing standard errors for the difficulty and discrimination parameters? Post your output here if you can, but you might consider examining those items for miscoding, and maybe even consider them for deletion. I'm not sure what else to advise.

If you have Stata 16, you can manually limit the number of iterations, e.g.

irt 2pl q1 q2 q3 q4 ..., iterate(100)

I have a feeling that 100 iterations should be sufficient to uncover any convergence issues.

Last, I forget exactly how sample size affects identification in SEM or similar models (and IRT is a type of SEM model). If you have very few cases, I think that might be a problem (but I'm not 100% sure on this).

[surendra kumar labanya]

Dear Sir.
Thank you so much for your kindly responses. My sample size exceeds seven deficiencies. When I run at the national level, I get the results occasionally. Similarly, when I start running at the state level, I get the results occasionally as well, but in a few states, some times I don't get the results at all. So i don't think it's a coding problem. But I have not worked with STATA 17. Since I do not have it. However, I am using STATA 14.2 and have also checked with STATA 16. The issues are the same for STATA 14.2 and 16. Hence, as per your kind suggestions, i am attaching the output files. So kindly go through the attached files. Thank you once again, and i' m looking forward to your positive responses.

With Best Regards,

Surendra

[Weiwen Ng]

As stated in the FAQ, we ask people to type out the exact code they are using, and also to paste results into code delimiters. Not everyone has the means to read xlsx or smcl files - they may be on mobile devices, they may have institutional restrictions, etc.

That said, your log file indicates that you typed this command:

irt 2pl math3 if state_code=23

Many of us remember the algebra problems where they give you two equations with two unknown variables and you have to solve for the two unknowns. If they gave you two equations and three unknowns, you wouldn't be able to identify unique values of x, y, and z that solve the equations. There is not enough information. Identification in structural equation models is a bit like that. You are trying to estimate math ability with one question. That's not enough information. Your model is not identified. The solution is simple: add more questions if you have them.

I don't know what is the absolute minimum number of items to have an identified model. In traditional linear SEM, it might be three questions (you're looking at the case where there's one latent variable and no other variables; in linear SEM you are, I believe, estimating a slope and intercept for each item, which is the same number of parameters as in a 2PL model). But only three questions won't let you discriminate very well among people of different reading levels. Basically, you need to enter more questions into the IRT model. 3 at the very least.

What if you only have one math question? Well, if you are doing any regression work, you could simply use that question as the dependent variable.

[surendra kumar labanya]

Dear Sir,
As I previously stated in the comments box, I occasionally get results at the national and sub-national levels. I do not face any problems there at all, but am facing problems only in a few states. If the data set is wrong then it should not show any results at national level analysis. So kindly find a solution to help us that will be better. Why is it happening ? Also, that would be beneficial to many more researchers like me and those who are using STATA to run the IRT 2PL method. This problem is not only with me. The same issues was previously also posted and they are also facing the same problems.

Thanking You,
Surendra

[Weiwen Ng]

I went and checked my own dataset by fitting an IRT model to one question at a time. It does actually converge in most cases. However, when it converges, the discrimination and difficulty parameters are very different from when I fit a full model with all the questions. Second, the model doesn't converge for a few of the questions, and there's no obvious reason why - in Stata 17, the program automatically constrained my discrimination parameter for me. Maybe a 2PL model is not intrinsically unidentified with only one question, but I would question the results. I'd also question the utility of the model - the whole point of IRT is to use responses to a group of questions to estimate the level of a latent trait. If you have only one question, then the answer to that question is your estimate, no fancy model needed.

The problem with convergence issues is that it's often difficult to say why the model doesn't converge. Your overall sample size is probably adequate. As I said before, if you are actually trying to fit an IRT model with only one question in it, I think this is ill advised and it may not end well. I have a gut feeling that adding more questions will make convergence issues less likely to occur.

As always, you can fit a proper model and present your results here and see if anyone has any insight. The output you showed had only one item in the model, so as I stated I am very skeptical to begin with. In this particular case, we can see that when the estimation terminated, the difficulty parameter was -0.7 or so, which is a number you'd see in real life, but the discrimination parameter was 837. That's completely unrealistic. I've seen numbers as high as 4 in some studies (symptom scales in health services research). 837 is not realistic. I'd examine the descriptive statistics. Maybe tabulate the proportion of correct answers by state. I did experiment, and it does look like with 2PL models, Stata will detect if an item does not vary at all within the estimation sample and return an error rather than start estimation.

[surendra kumar labanya]

Dear Sir,
Thank you so much for your kind response. Now i am considering two variables and ran the regression. I am attaching the results. Kindly go through them. The results of discrimination are not acceptable. To get the next results, it just takes time and there is no convergence at all. Same code, nothing is changed. Only the state code changed. But if I run only for reading, I am getting good results for state code 36, but I will not get the results for state code 8. So, please assist me in resolving this matter.

irt 2pl read arithmetic if state_code==36

Fitting fixed-effects model:

Iteration 0: log likelihood = -5201.4542
Iteration 1: log likelihood = -5195.1732
Iteration 2: log likelihood = -5195.1686
Iteration 3: log likelihood = -5195.1686

Fitting full model:

Iteration 0: log likelihood = -4740.2236
Iteration 1: log likelihood = -3598.0782
Iteration 2: log likelihood = -2886.1185 (not concave)
Iteration 3: log likelihood = -3199.5163 (not concave)
Iteration 4: log likelihood = -3397.1565 (not concave)
Iteration 5: log likelihood = -2842.0443 (not concave)
Iteration 6: log likelihood = -3073.9297 (not concave)
Iteration 7: log likelihood = -3083.3215 (not concave)
Iteration 8: log likelihood = -3844.2968 (not concave)
Iteration 9: log likelihood = -3897.8537 (not concave)
Iteration 10: log likelihood = -2980.3766 (not concave)
Iteration 11: log likelihood = -3072.0012 (not concave)
Iteration 12: log likelihood = -3072.5612
Iteration 13: log likelihood = -2822.5961
Iteration 14: log likelihood = -2678.8609
Iteration 15: log likelihood = -2671.4107
Iteration 16: log likelihood = -2672.9273
Iteration 17: log likelihood = -2675.5083
Iteration 18: log likelihood = -2705.8059
Iteration 19: log likelihood = -2713.4841
Iteration 20: log likelihood = -2708.4471
Iteration 21: log likelihood = -2701.4452
Iteration 22: log likelihood = -2699.07
Iteration 23: log likelihood = -2699.5753
Iteration 24: log likelihood = -2700.7915
Iteration 25: log likelihood = -2702.1852
Iteration 26: log likelihood = -2703.0041
Iteration 27: log likelihood = -2703.5295
Iteration 28: log likelihood = -2703.7806
Iteration 29: log likelihood = -2703.9113
Iteration 30: log likelihood = -2703.9703
Iteration 31: log likelihood = -2703.9903
Iteration 32: log likelihood = -2703.9908
Iteration 33: log likelihood = -2703.9824
Iteration 34: log likelihood = -2703.9707
Iteration 35: log likelihood = -2703.9586
Iteration 36: log likelihood = -2703.9472
Iteration 37: log likelihood = -2703.9371
Iteration 38: log likelihood = -2703.9285
Iteration 39: log likelihood = -2703.9212
Iteration 40: log likelihood = -2703.9151
Iteration 41: log likelihood = -2703.91
Iteration 42: log likelihood = -2703.9058
Iteration 43: log likelihood = -2703.9024
Iteration 44: log likelihood = -2703.8996
Iteration 45: log likelihood = -2703.8973
Iteration 46: log likelihood = -2703.8954
Iteration 47: log likelihood = -2703.8938
Iteration 48: log likelihood = -2703.8925
Iteration 49: log likelihood = -2703.8915
Iteration 50: log likelihood = -2703.8906
Iteration 51: log likelihood = -2703.8899
Iteration 52: log likelihood = -2703.8894
Iteration 53: log likelihood = -2703.8889
Iteration 54: log likelihood = -2703.8885
Iteration 55: log likelihood = -2703.8882
Iteration 56: log likelihood = -2703.8879

Two-parameter logistic model Number of obs = 5,038
Log likelihood = -2703.8879
------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
read |Discrim | 107.5989 16.95238 6.35 0.000 74.37282 140.8249
Diff | -.8040903 .004191 -191.86 0.000 -.8123045 -.7958761
-------------+----------------------------------------------------------------
arithmetic | Discrim | 74.86356 6.581697 11.37 0.000 61.96367 87.76345
Diff | -.8274752 .0025848 -320.14 0.000 -.8325412 -.8224091
------------------------------------------------------------------------------

. irt 2pl read arithmetic if state_code==8

Fitting fixed-effects model:

Iteration 0: log likelihood = -29087.336
Iteration 1: log likelihood = -29011.356
Iteration 2: log likelihood = -29011.298
Iteration 3: log likelihood = -29011.298

Fitting full model:

Iteration 0: log likelihood = -26348.445
Iteration 1: log likelihood = -19536.818
Iteration 2: log likelihood = -15729.981 (not concave)
Iteration 3: log likelihood = -20273.637 (not concave)
Iteration 4: log likelihood = -18457.705 (not concave)
Iteration 5: log likelihood = -16252.285 (not concave)
Iteration 6: log likelihood = -17997.848 (not concave)
Iteration 7: log likelihood = -18281.489 (not concave)
Iteration 8: log likelihood = -16786.25 (not concave)
Iteration 9: log likelihood = -22802.106 (not concave)
Iteration 10: log likelihood = -18583.464 (not concave)
Iteration 11: log likelihood = -23007.687 (not concave)
Iteration 12: log likelihood = -16680.55 (not concave)
Iteration 13: log likelihood = -23453.86 (not concave)
Iteration 14: log likelihood = -18473.667 (not concave)
Iteration 15: log likelihood = -22600.668 (not concave)
Iteration 16: log likelihood = -22795.811 (not concave)
Iteration 17: log likelihood = -19267.541
Iteration 18: log likelihood = -25875.988 (not concave)
Iteration 19: log likelihood = -20236.499 (not concave)
Iteration 20: log likelihood = -22341.647 (not concave)
Iteration 21: log likelihood = -19920.683 (not concave)
Iteration 22: log likelihood = -22886.157 (not concave)
Iteration 23: log likelihood = -16692.6 (not concave)
Iteration 24: log likelihood = -21813.156 (not concave)
Iteration 25: log likelihood = -18809.165 (not concave)
Iteration 26: log likelihood = -23408.529 (not concave)
Iteration 27: log likelihood = -18608.246 (not concave)
Iteration 28: log likelihood = -22364.588 (not concave)
Iteration 29: log likelihood = -16732.223 (not concave)
Iteration 30: log likelihood = -22768.043 (not concave)
Iteration 31: log likelihood = -17471.162 (not concave)
Iteration 32: log likelihood = -23017.224 (not concave)
Iteration 33: log likelihood = -18103.511 (not concave)
Iteration 34: log likelihood = -22834.618 (not concave)
Iteration 35: log likelihood = -18175.095 (not concave)
Iteration 36: log likelihood = -21730.724 (not concave)
Iteration 37: log likelihood = -18438.991 (not concave)
Iteration 38: log likelihood = -21936.084 (not concave)
Iteration 39: log likelihood = -18722.868 (not concave)
Iteration 40: log likelihood = -24413.117 (not concave)
Iteration 41: log likelihood = -20541.681 (not concave)
--Break--



Thanking you,

Surendra

[Weiwen Ng]

As stated earlier, you need to add more items to the model. I am not willing to trust the results from an IRT model with only 1 or 2 items for reasons I explained in post #6.

Also, in education, I'm under the impression that reading and math ability are usually modeled as separate traits. I am not sure how justified it is to add one arithmetic item to a model with one reading item.

[surendra kumar labanya]

Dear Sir,
One thing i can't understand is that if you are talking about more than one variable, it should be included. I completely agree with your views. But for a few states, I am getting good results with one variable and the results are acceptable. So I can say it is not about one variable or two variables problem, sir. Therefore, please find the expert in this area and help us.

Best regards,
Surendra

[Weiwen Ng]

...
One thing i can't understand is that if you are talking about more than one variable, it should be included. I completely agree with your views. But for a few states, I am getting good results with one variable and the results are acceptable. So I can say it is not about one variable or two variables problem, sir. Therefore, please find the expert in this area and help us.

...

If you completely agree with my views and you have multiple (and I mean more than two!) items, and preferably items measuring the same academic ability (e.g. all math OR all verbal), then why don't you try including all the math items that you have in a 2PL model first?

Even if the model converges, I urge you to read Kenny's page, and also this page, on model identification carefully. Read under measurement model. Ignore the bits about correlated errors, because those aren't applicable in IRT. In traditional SEM, you are estimating 2 parameters per item, and you fix the first item's loading (equiv. to discrimination) at 1. In a 2PL model, you are also estimating 2 parameters per item, and you don't fix the first item loading but you do fix the latent variable variance at 1. Hence, I believe that the rules on identification should apply, and Kenny says you need at least 3 items per latent trait to identify the model. Hence, unless someone knowledgeable tells me otherwise, your model is not identified. I'm surprised that it converged as often as it did, but I would not trust the results even if it converged. If you don't grasp the importance of identification, this is what Kenny says, emphasis mine:

A model is said to be identified if there exists a unique solution for all of the model's parameters. If there is no unique solution, then the model is of little value. Note that a model may not be identified, but some of the model’s parameters may be identified.