Please note that a version of this same question was posted to the Medstats group.
A colleague has paired scores for n = 252 subjects (assume interval scale measurement). The subjects are children aged 12-18. The sample size by year ranges from 12 to 40. One of the paired scores was provided by the child, and the other by a parent. The child-parent pairs are all independent. My colleague wishes to know if the level of agreement between child & parent depends on the age of the child.
Not unexpectedly, one of the suggestions in the Medstats forum was to "estimate the limits of agreement as a function of age using the half-Normal regression method in Bland & Altman (1999), (Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8, 135-160). For those who don't know it, the Bland-Altman limits of agreement approach is based on paired difference scores. And as I noted in my reply to that suggestion, my colleague has been specifically steered away from using difference scores by a peer reviewer, who referred her to this article: One thought I had was to estimate a model (using -mixed-) that examines the interaction between rater (child, parent) and age of the child. That would only tell me whether the mean difference between children and parents varies by age of the child. But the mean difference between raters is only one component of agreement, the other being the degree of linear association. It seems to me, therefore, that it would be better to compute the ICC for each age separately. I think the test of heterogeneity from a meta-analysis of those ICCs would address quite directly my colleague's question. (It also occurs to me that one could use -metareg- if the forest plot suggests that the ICCs appear to be a linear function of age.)
So one question is whether a test of heterogeneity of the ICCs for each age group makes sense. If you think it does, can you point to any resources on meta-analyzing ICCs? E.g., should the r-to-Z transformation be used, like it is for meta-analysis of Pearson correlations? If so, does 1/sqrt(n-3) still give a reasonable estimate of the SE? Or should the ICCs be meta-analyed directly? If you think this is the way to go, there's a problem in that -icc- does not (I think) report a SE. I could instead use -mixed- with the reml option to get a SE, as in this old thread. But note that the CI is not the same as the CI from -icc-.
Alternatively, perhaps you suggest another approach to addressing my colleague's question, bearing in mind that the raw data are available, and that she wishes to avoid analysis of difference scores. It's entirely possible I've overlooked a much better approach.
Sorry about the length of this post. I just wanted to provide sufficient context.
Cheers,
Bruce
A colleague has paired scores for n = 252 subjects (assume interval scale measurement). The subjects are children aged 12-18. The sample size by year ranges from 12 to 40. One of the paired scores was provided by the child, and the other by a parent. The child-parent pairs are all independent. My colleague wishes to know if the level of agreement between child & parent depends on the age of the child.
Not unexpectedly, one of the suggestions in the Medstats forum was to "estimate the limits of agreement as a function of age using the half-Normal regression method in Bland & Altman (1999), (Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8, 135-160). For those who don't know it, the Bland-Altman limits of agreement approach is based on paired difference scores. And as I noted in my reply to that suggestion, my colleague has been specifically steered away from using difference scores by a peer reviewer, who referred her to this article: One thought I had was to estimate a model (using -mixed-) that examines the interaction between rater (child, parent) and age of the child. That would only tell me whether the mean difference between children and parents varies by age of the child. But the mean difference between raters is only one component of agreement, the other being the degree of linear association. It seems to me, therefore, that it would be better to compute the ICC for each age separately. I think the test of heterogeneity from a meta-analysis of those ICCs would address quite directly my colleague's question. (It also occurs to me that one could use -metareg- if the forest plot suggests that the ICCs appear to be a linear function of age.)
So one question is whether a test of heterogeneity of the ICCs for each age group makes sense. If you think it does, can you point to any resources on meta-analyzing ICCs? E.g., should the r-to-Z transformation be used, like it is for meta-analysis of Pearson correlations? If so, does 1/sqrt(n-3) still give a reasonable estimate of the SE? Or should the ICCs be meta-analyed directly? If you think this is the way to go, there's a problem in that -icc- does not (I think) report a SE. I could instead use -mixed- with the reml option to get a SE, as in this old thread. But note that the CI is not the same as the CI from -icc-.
Alternatively, perhaps you suggest another approach to addressing my colleague's question, bearing in mind that the raw data are available, and that she wishes to avoid analysis of difference scores. It's entirely possible I've overlooked a much better approach.
Sorry about the length of this post. I just wanted to provide sufficient context.
Cheers,
Bruce
Comment