Factor analysis or similar data reduction technique for variables with different response options (binary and multiple response options)

[Binita Subedi]

I am trying to create a variable that assess the construct of "knowledge" using participant's responses to about 20 questions. These questions have different response options. Some of the questions have Yes/No options, while others have multiple response options where multiple answers are possible. Among the multiple response options, two of the options are "Other(specify)" and "Don't know". Other options in the multiple response questions range in number from 4 to 7 for different questions. I was wondering if the responses to these different questions can be combined to form one variable using something like factor analysis.

An example multiple response question would be:
What are some potential advantages of composting/?
Option 1 - Advantage one
Option 2 - Advantage two
Option 3 - Advantage three
Option 4 - Advantage four
Option 5 - Other(specify)
Option 6 - Don't know

I also want to know if anything changes if I am only using 4 questions/items as opposed to 20 to form the variable.

[Clyde Schechter]

Well, if this construct is truly knowledge, then there must be correct and incorrect responses to all of the items. A simple way to score that construct would be to create, for each item, a new variable that takes the value 1 if the response given is correct and 0 if incorrect or omitted. The total of those new 0/1 variables would be a knowledge score. There are, of course, more sophisticated approaches that rely on more advanced psychometric techniques, but I'm guessing you don't want to go there.

[Binita Subedi]

Thank you for your response Clyde. Just to confirm that I am getting this correct, if there are 4 correct response options for one question/item, then I will be creating 4 binary variables for that one question. Then I will do the same for all questions, and sum up the scores.

I might not be able to do the advanced techniques, but could you just name them for future reference?

Another related question that I have is -
In the same questionnaire, the same respondents have answered "practice" questions which are of similar nature to the "knowledge" questions in that they have a mix of binary (yes/no) and ordinal/multiple correct response options.
If I need to compare the "knowledge" scores with the "practice" scores for the same individuals, is it necessary that the maximum total scores possible be equal for the "knowledge" and "practice" scores? Is Wilcoxon signed rank test suitable to see if the practice matches the knowledge?

[Clyde Schechter]

Well, I actually imagined that each question had only one correct response and that all others were distractors. But even when there are multiple correct responses, you can modify the approach. You then need a new 0/1 variable for each possible response. You would then score 1 for each correct response selected and also 1 for each incorrect response not selected. Correct responses not selected and incorrect responses selected score 0. Add up the scores for each response.

I don't see how you can extend this approach to "practice" scores, however, because it depends on there being "correct" and "incorrect" responses, which would not be the case for practices.

The more advanced approach I had in mind is to use item response theory (IRT). But this data now does not sound very well suited to it, though I can't entirely rule it out.

Assuming you do manage to figure out a way to score your practice items, I do not think it makes sense conceptually to "compare" knowledge and practice in the way you describe. The Wilcoxon signed rank test is a test of the equality of the scores. But equality would only make sense if the maximum total score possible were equal for both knowledge and practice. And even then, it would only make sense if the knowledge and practice items were equally difficult. Assessing the difficulty of items would require the use of IRT, which in turn would enable you to find equivalent scores on the two metrics. Stata has a whole suite of commands for IRT, but if you are not familiar with the underlying psychometrics, it would be unwise to go there. Moreover, as I say, I have some doubts whether this kind of data lends itself to IRT analysis.

[Binita Subedi]

Thank you again for your response. This has really been helpful to clear things up.

I will propose a few different ways that I think I could go about the analysis. Could you please comment on them?

1. Run a principal component analysis on the knowledge questions and on practice questions. Compare the component scores with Wilcoxon signed rank test or paired t test. I am not sure if paired t test works with factor scores.
2. The knowledge questions as well as the practice questions have different dimensions to them - for example - some questions relate to handwashing, some relate to poultry management, some relate to nutrition, and so on. I want to compare knowledge and practices on these dimensions separately, without generating one knowledge variable and one practice variable and regress practice on knowledge. Would it be wise to run a regression like this?
3. Would it be better to come up with cut-off points based on the data for the knowledge variable, for example, by using the mean or the median to categorize those with good and poor knowledge, and use those categories instead of the scores? This might not make sense for the practice variable.
4. Or some combination of the above

I would really appreciate your guidance on this. I have searched and searched for some way out and have not found much help elsewhere.

[Clyde Schechter]

1. Run a principal component analysis on the knowledge questions and on practice questions. Compare the component scores with Wilcoxon signed rank test or paired t test. I am not sure if paired t test works with factor scores.

The questions don't lend themselves to this kind of analysis because of the presence of multiple correct response to a question. So, at best, you would have to do this on the correct/incorrect 1/0 variables assigned to each response. When you then take a first principal component you are throwing away part of the information in the data--just how much will depend on the proportion of variance accounted for by the first component. You might be discarding a lot, or maybe only a little. Comparing the first components with a Wilcoxon signed rank or paired t-test still makes no sense conceptually. Without some measure of the "difficulty" of the items in each construct, it is impossible to interpret the findings. Let's take a clearer example. Suppose we test a bunch of high school students in algebra and in chemistry. What does it mean if the mean score on the algebra test is 80 and the mean score on the chemistry test is 70? Impossible to say: maybe the students actually know the chemistry test better, but the chem test was hard and the algebra test was very easy. Do you see the problem? Factor analysis or principal components does not get around this.

2. The knowledge questions as well as the practice questions have different dimensions to them - for example - some questions relate to handwashing, some relate to poultry management, some relate to nutrition, and so on. I want to compare knowledge and practices on these dimensions separately, without generating one knowledge variable and one practice variable and regress practice on knowledge. Would it be wise to run a regression like this?

In essence, this takes the original problem and decomposes it into a set of several problems, each of which is a piece of the larger one. But these pieces are just as intractable as the original problem. All of the objections to the original problem apply equally to the dimension-specific problems. And, to add to the difficulty, these dimensions are, of course, based on a smaller number of items/responses, so that they are noisier in the first place!

The part of this approach that I do endorse is relating the knowledge and practice scores by regressing them. Actually, before doing that, I would do scatterplots to see if it looks like a linear relationship is appropriate. If so, a regression is a good way to quantify that. If not, you will be able to see if you can relate them linearly after some kind of transformation. And I would consider applying this graphics/regression approach to the combined knowledge and combined practice scores as well. On balance I think this is the best of these approaches.

3. Would it be better to come up with cut-off points based on the data for the knowledge variable, for example, by using the mean or the median to categorize those with good and poor knowledge, and use those categories instead of the scores? This might not make sense for the practice variable.

This is a truly terrible idea. Imposing cutpoints on count or continuous variables makes them noisier and discards information. If, say, you use a cutpoint of 5 correct answers, you are saying that a person who scores 5 is radically different from a person who scores 4, but is the same as a person who scores 20. That's almost always nonsense. Unless there is some evidence, external to the data, that justifies a certain cutpoint as distinguishing things that are actually different in some important, real-world way, this is just data mutilation.

[Binita Subedi]

Thank you for your response Clyde. In your last response, you recommended against using PCA or factor analysis for questions with multiple correct responses. Would it make sense to use PCA on a Likert scale type questions where a set of 8 questions are assessing quality of interaction on a scale of always, often, sometimes, rarely, never? I have seen PCA used for these kind of questions, but still not very clear when to use PCA and when to just use an unweighted average. I have found that things like wealth index are always calculated using PCAs or factor analysis, while they are not mostly used for other variables with similar question structures except to validate the scales. Are there instances where PCAs or factor analyses not ideal even with likert scale items or binary variables?

My second question is - Can PCAs or factor analyses be used to combine likert scale items and binary variables together? In my dataset, I have quality of interaction assessed using both likert scale items and binary responses, which I wanted to combine into one or more component scores and use them for regression.