Test Retest (ICC) reliability with missing data

[Mark Wilberforce]

Hello

I am working with a test and retest of my schedule of 30 items. I will be summing them to form a total score, which will be the subject of an ICC calculation. At least in part because of the population I am working with, I have a lot of missing data for this postal questionnaire. For 101 administrations of my schedule only 59 have complete data at both administrations. It's a lot of data to throw away (and will result in bias).

I'm trying to think of a way of running my ICC calculation without relying on complete cases only.

I have a few ideas, but before I run away with them, I wondered if fellow statausers might have some insights. Or can point me the right way.

Thank you

Mark Wilberforce
PhD candidate, University of Manchester, UK

[Joseph Coveney]

If you're going to use a sum score, then have you considered using the arithmetic average of available item responses for each occasion?

Many of the questionnaires that I've dealt with have a threshold of nonmissing data for a (sub)scale—for example, not more than two items out of a total of five items may have no response or have an invalid response—or else the score is set to missing. So, you might want to consider something similar, perhaps a majority of items with a response on the occasion, and if so, then use the average of the available responses × 30 for the prorated score for that administration.

[Mark Wilberforce]

Hi Joseph - thank you very much. Yes, I had thought of the average score but am glad to have that suggested by you because I wasn't sure why I hadn't seen it to be common practice. I think simply removing cases from the ICC calculation will bias the reliability estimates (presuming those with missing score are less reliable on other questions).

I will look at some thresholds too - and generally present different scenarios.

Thank you.

Mark