Examining the stability of an ordinal variable over time.

[Andrea Burgess]

Hi,
I have a longitudinal study where participants were seen between 1 and 6 times at set appointment times of 18, 24, 30, 36, 48, 60 months (the age as continous variable is also available). I am interested to see if there is change over time in the ordinal variable macs (1to5). MACS is not normal distribution.

----------------------- copy starting from the next line -----------------------

* Example generated by -dataex-. To install: ssc install dataex
clear
input int patientid byte appt_type float age byte macs
1386 24 27.02 1
1386 30 32.09 1
1386 36 37.38 2
1386 48 48.46 1
1386 60 63.39 1
1394 24 23.41 2
1394 30 29.65 2
1394 36 35.41 2
1394 48 47.61 2
1394 60  60.2 1
1396 24 22.09 2
1396 30    29 2
1396 36 36.03 2
1396 48 47.38 1
1396 60 63.06 1
1417 30 18.20 4
1417 36 36.79 4
1417 48  48.3 3
1417 60 61.71 2
1427 18 19.79 5
end
label values appt_type appt_type_labels
label def appt_type_labels 18 "18 months", modify
label def appt_type_labels 24 "24 months", modify
label def appt_type_labels 30 "30 months", modify
label def appt_type_labels 36 "36 months", modify
label def appt_type_labels 48 "48 months", modify
label def appt_type_labels 60 "60 months", modify
label values macs macs_labels
label def macs_labels 1 "MACS 1", modify
label def macs_labels 2 "MACS 2", modify

------------------ copy up to and including the previous line ------------------


From my first research, I looked at change over time using long format and used

mean macs, over(appt_type) lincom _subpop_1 - _subpop_2, or lincom _subpop_2 - _subpop_3, or lincom _subpop_3 - _subpop_4, or lincom _subpop_4 - _subpop_5, or lincom _subpop_5 - _subpop_6, or lincom _subpop_1 - _subpop_6, or lincom _subpop_1 - _subpop_3, or lincom _subpop_1 - _subpop_3, or

this showed significant difference in mean macs between 18 months and 24 months but not between other groups

I also looked at it in wide format and used

 
 gen macsdiff2= macs24 - macs18 gen macsdiff3 = macs30 - macs24 gen macsdiff4 = macs36 - macs30 gen macsdiff5 = macs48 - macs36 gen macsdiff6 = macs 60- macs48 ttest macsdiff2=0 ttest macsdiff3=0 ttest macsdiff4=0 ttest macsdiff5=0 ttest macsdiff6=0

This gave same result as lincom with significant diff between 18 and 24 months only, showing that there was stability in macs between 30-60 months.
This was the result which I was expecting.

However, it was pointed out to me that macs was not normally distributed, and ordinal (5 level).

Then I was not sure if this is correct (even though it made sense), and so have used
Kappa statistic with the result that there is significant agreement between macs at all ages (including macs18 and macs24)

 
 kap macs18 macs24

Expected
Agreement Agreement Kappa Std. Err. Z Prob>Z
-----------------------------------------------------------------
63.33% 22.06% 0.5296 0.0661 8.01 0.0000

 
 kap macs24 macs30

Expected
Agreement Agreement Kappa Std. Err. Z Prob>Z
-----------------------------------------------------------------
78.76% 26.91% 0.7094 0.0530 13.39 0.0000

kap macs18 macs24, wgt(w)

Ratings weighted by:
1.0000 0.7500 0.5000 0.2500 0.0000
0.7500 1.0000 0.7500 0.5000 0.2500
0.5000 0.7500 1.0000 0.7500 0.5000
0.2500 0.5000 0.7500 1.0000 0.7500
0.0000 0.2500 0.5000 0.7500 1.0000

Expected Agreement Agreement Kappa Std. Err. Z Prob>Z
-----------------------------------------------------------------
90.42% 62.43% 0.7449 0.0872 8.54 0.0000




I also used Friedman test to look at the difference….

 
  friedman macs18 macs24

Friedman = 105.6730
Kendall = 0.8955
P-value = 0.0002

 
 friedman macs24 macs30

Friedman = 193.4080
Kendall = 0.8634
P-value = 0.0000

 
 friedman macs18 macs24 macs30

Friedman = 124.0727
Kendall = 0.8616
P-value = 0.0000

 
 friedman macs18 macs24 macs30 macs48 macs60

Friedman = 175.0047
Kendall = 0.8334
P-value = 0.0000






So, now I am thoroughly confused and am seeking guidance on what is best way to look at stability over time in macs.

Kind regards,
Andrea Burgess.

[Carlo Lazzaro]

Andrea:
what about -xtoprobit-?

[Andrea Burgess]

Hi Carlo,

I don't know xtoprobit - have just looked it up in the help.

When I tried it just then, I get the message:

Iteration 0:   log likelihood = -1266.1353  
cannot compute an improvement -- discontinuous region encountered
r(430);

[Carlo Lazzaro]

Andrea:
how many observations are included in your sample?

[Clyde Schechter]

Well, I think you need to clarify just what you mean by "stability" over time. The first question that comes to my mind is whether you are concerned about a trend in a particular direction over time, or whether you are concerned more generally even about haphazard up-and-down fluctuations. The approaches to recognizing those two types of instability would be different. You also need to consider that there will necessarily be a certain amount of bouncing around in any variable that is subject to measurement error (which, realistically, is all variables). So only changes of a magnitude that goes beyond measurement error should be of concern. But it may be that measurement error is very small, and even much larger changes than that would not be considered instability. So you need to identify just how big a change would be considered instability. Once you get clarity on these questions, it will be easier to identify a way to look for it in the data.

[Weiwen Ng]

Hello, Andrea, I remember this discussion from last time.

Just to clarify, we want the residuals from a regression to be normal, or at least normal enough. The distribution of the outcome variable itself doesn't necessarily have to be normal. That said, because macs is ordinal, the logical thing would be to use an ordinal regression, either logit or probit, for the analysis. I see that xtoprobit won't converge. Have you tried xtologit?


Kappa is a statistic that may not account for the fact that you have repeated measures on the same individual, and also that the same clinician might be rating macs at each visit. Not sure I'd use that for a main analysis. The question is, what are you trying to do here? In your other question, you said your main outcome was a self care scale.

If you're doing a separate analysis on macs, then it would be best to do ordered logit or probit. If you're just doing something for descriptive statistics, then really, a t-test for a difference in means may well be acceptable. It fits with what I've seen in healthcare research. Yes, we know that model is wrong, but all models are wrong, and this one is useful enough for table 1.

[Joseph Coveney]

I take it that you're trying to follow along this. They ended up just using ANOVA for their intraclass correlation; I tried to use xtoprobit on the data in their Table 3, and I get the same convergence problem that you do with your data (see attached DO file). The counts are quite peaked along the diagonal, and there are too many transitions (e.g., MACS V ↔ MACS I) with zero counts.

You might have to go with something like

oprobit macs i.appt_type, cluster(patientid)

and perhaps assess "stability" in terms of confidence bounds (à la bioequivalence testing) of the coefficients at successive appointment times, but I agree with Clyde that you'll first need to clarify just what "stability" of the MACS level means to you and your colleagues.

[Andrea Burgess]

Hi,
Thanks for your responses.
Sample size is n=291 with 1065 observations. The sample size differs at different time points eg. n=75 at 18 months and n=131 at 24 months. n=242 60 months

Overall I am looking at self-care trajectories (continuous variable), according to manual ability classification (macs). However, I want to examine if macs is stable over time. There were about 3 raters doing all these participants, but I am not looking at the interrater reliability. I am looking at if the classification system was stable over time. eg. is a child who is classified at macs level 2 when they are 18 months old, still rated as macs level 2 when they are 5 years old ?
The classification system was not really designed with such young children in mind, and so the fact that the participants rating may change with age, just because they are older is not unexpected. However it could be stable.
I will look at the measurement error now and hopefully get back to you soon.
Regards, Andrea.

[Carlo Lazzaro]

Andrea:
if you -xtset- you data, you can get a rough idea of what went on via -xttrans-.

[Andrea Burgess]

Hi
Yes - I have used the following commands, and am just not sure how to put it all together. eg. how do the xtsum "within difference" and the xttab "within percents" relate to each other ?

xtsum macs - got the within sd (0.33) and between sd (1.3)

xttab macs - . The total within percent = 70.46, ("within percent is the normalized between weighted average of the within percents..It is a measure of the overall stability of the - variable." Should this be what I report ?

xttrans macs, freq - this is a great way to see the movement across the macs levels and shows which is the most stable of the classification levels. eg. most stable at levels 1 and 5, with more movement in the middle levels.

But how should I measure the changes (which I can see in that transition table) - to see if these changes are significant? I guess one thing that the xttrans table doesn't do is show when the transitions occurred.
According to the means (t tests), the biggest change occurs between 18 and 24 months.

I have trialled the suggestion from Joseph (thank you) - oprobit command - but I am not at all sure about interpreting the results.

Ordered probit regression                       Number of obs     =      1,065
                                                Wald chi2(5)      =      28.83
                                                Prob > chi2       =     0.0000
Log pseudolikelihood = -1494.2789               Pseudo R2         =     0.0066
 
                            (Std. Err. adjusted for 291 clusters in patientid)
------------------------------------------------------------------------------
             |               Robust
        macs |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
   appt_type |
  24 months  |  -.3054892   .0998161    -3.06   0.002    -.5011252   -.1098532
  30 months  |  -.4117731    .108612    -3.79   0.000    -.6246487   -.1988975
  36 months  |  -.4465379   .1041684    -4.29   0.000    -.6507041   -.2423716
  48 months  |  -.5074192   .1078668    -4.70   0.000    -.7188341   -.2960042
  60 months  |  -.5852252   .1109432    -5.27   0.000      -.80267   -.3677804
-------------+----------------------------------------------------------------
       /cut1 |   -.746849   .1122081                     -.9667729   -.5269251
       /cut2 |   .0948515   .1192142                      -.138804     .328507
       /cut3 |   .3898104   .1224188                      .1498739    .6297468
       /cut4 |   .6692395    .128068                      .4182309    .9202482
------------------------------------------------------------------------------
 

I really appreciate all of your help and comments.

Regards,
Andrea.

[Carlo Lazzaro]

Andrea:
see -oprobit postestimation- enry in Stata .pdf manual.
That said, it seems that there is a time trend reduction in the value of the -depvar-.
I would try:

testparm i.appt_type

to investigate that issue-

[Andrea Burgess]

Sorry for delay in getting back -

 testparm i.appt_type

results was that there was a significant difference for macs at the different time points. chi2( 5) = 28.83 Prob > chi2 = 0.0000

When I looked at differences between individual time points

test 18.appt_type = 24.appt_type

This also showed significant difference. chi2( 1) = 9.37 Prob > chi2 = 0.0022
This fits in with my Friedman test which showed signifcant differences as well.

Would I interpret the result for the ordered probit regression (seen in table above): An appt_type at 24 months, versus 18 months (the reference group), decreases the z-score by 0.305. ?

What I am confused about though, is if you use kappa statistics - you get opposite results, and same with ICC it is 0.92.


As Joseph pointed out there are some similar papers which use ICC, McNemar's, Kappa, Friedman. And I have been trialling out all of them.
Is the difference in results (Friedman & oprobit Vs Kappa, ICC) because of the suitability of the measure in measuring an ordinal variable ?

Regards,
Andrea.

[Clyde Schechter]

I've dropped out of this thread for a while. But I'll try to reassert the argument I made in #5.

Statistical significance is a highly overrated concept. It is widely misunderstood, and when properly understood, often not useful. See Wasserstein RL & Lazar NA. The ASA's statement on p-values: context, process, and purpose. The American Statistician 2016. You can link to it at http://dx.doi.org/10.1080/00031305.2016.1154108. And when you have time, read all the other articles that accompanied that summary statement. The most important general consideration in your particular context here is that statistical significance and p-values are statistics that are calculated conditional on some particular selected model of the data, and that they are also sensitive to sample size. As such, the "objectivity" that is often attributed to statistical significance is nothing more than an illusion: the choice of the underlying model is entirely based on judgment, and the p-value flows from that. You are experiencing exactly that in your various different "tests" giving you apparently conflicting answers. But they are not really conflicting answers: they are answers to different questions. You haven't yet figured out what question you actually care about. My hunch is that, in fact, there is no reason to care about any of the particular questions answered by the tests you have undertaken so far.

General considerations about p-values aside, I think you will continue to struggle with, and be bemused by, this problem until you abandon the quest for statistical significance and define for yourself, perhaps with the help of your colleagues, a notion of pragmatic or clinical significance to apply here. After all, you did not set out in this research to test some specific hypothesis about the stability of the macs measure. The issue has come up simply because you are wondering whether it is a good enough measure to use as a predictor for a different hypothesis test.In fact, if the macs score were a variable with interval-level characteristics, you would probably just calculate the intra-class correlation and then make a judgment call as to whether it was high enough for your purposes. While you could calculate a p-value to see if the intra-class correlation is "statistically significant," that would be a pointless exercise: the null hypothesis of ICC = 0 is a ridiculous straw man. You would simply look at the observed value and decide. You are facing difficulty here only because the ordinal nature of the macs classification makes it questionably to apply the usual convenient statistics.

Looking closely at the output of -xttrans macs, freq- and pondering it is likely to point you in the right direction. Suppose you find that 1 or 2% of people in a certain category at one age move up or down one category at the next age. Would we not all agree that this is of no practical importance and that for the purposes to which we will put the macs measure, this is quite satisfactory? Yet such a small deviation from constancy could be found "statistically significant" in a sample the size of yours with a sufficiently rigid model. At the other extreme, if you found that 40% of all children shift categories from one measurement to the next, we could all agree that one could not much depend on the macs as an indicator of anything useful, at least not of anything that has some degree of permanence. Yet were you to apply a sufficiently accommodating model,in a small or even in a moderately large sample, it could very well not attain "statistical significance." I imagine that what you actually see in -xttrans- output is somewhere between those extremes. But that -xttrans- output is about as close to an unvarnished exposition of the changeability of the macs over time as it is possible to construct. As models go, it embodies few assumptions, and it is hard to even conceive of a notion of stability that could be operationalized without those minimalist assumptions..

[Joseph Coveney]

Is the difference in results (Friedman & oprobit Vs Kappa, ICC) because of the suitability of the measure in measuring an ordinal variable ?

No. It's because they look at different things. You can have very high ICC (even absolute) and still have differences between levels of the fixed effect.

.ÿversionÿ14.2

.ÿ
.ÿclearÿ*

.ÿsetÿmoreÿoff

.ÿsetÿseedÿ1375167

.ÿ
.ÿquietlyÿsetÿobsÿ50

.ÿgenerateÿbyteÿpidÿ=ÿ_n

.ÿgenerateÿdoubleÿuÿ=ÿ2ÿ*ÿrnormal()

.ÿ
.ÿforvaluesÿiÿ=ÿ1/3ÿ{
ÿÿ2.ÿÿÿÿÿÿÿÿÿgenerateÿdoubleÿsco`i'ÿ=ÿuÿ-ÿ`i'ÿ/ÿ10ÿ+ÿrnormal()ÿ/ÿ10
ÿÿ3.ÿ}

.ÿquietlyÿreshapeÿlongÿsco,ÿi(pid)ÿj(rid)

.ÿ
.ÿiccÿscoÿpidÿrid,ÿmixedÿtestvalue(0.999)

Intraclassÿcorrelations
Two-wayÿmixed-effectsÿmodel
Consistencyÿofÿagreement

Randomÿeffects:ÿpidÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿtargetsÿ=ÿÿÿÿÿÿÿÿ50
ÿFixedÿeffects:ÿridÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿratersÿÿ=ÿÿÿÿÿÿÿÿÿ3

--------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿscoÿ|ÿÿÿÿÿÿÿÿICCÿÿÿÿÿÿÿ[95%ÿConf.ÿInterval]
-----------------------+--------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿIndividualÿ|ÿÿÿ.9976364ÿÿÿÿÿÿÿÿ.996224ÿÿÿÿ.9985794
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿAverageÿ|ÿÿÿ.9992109ÿÿÿÿÿÿÿ.9987382ÿÿÿÿÿ.999526
--------------------------------------------------------------
Fÿtestÿthat
ÿÿICC(1)=1.00:ÿF(49.0,ÿ98.0)ÿ=ÿ0.42ÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿ=ÿ0.999
ÿÿICC(k)=1.00:ÿF(49.0,ÿ98.0)ÿ=ÿ1.27ÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿ=ÿ0.160

Note:ÿICCsÿestimateÿcorrelationsÿbetweenÿindividualÿmeasurements
ÿÿÿÿÿÿandÿbetweenÿaverageÿmeasurementsÿmadeÿonÿtheÿsameÿtarget.

.ÿiccÿscoÿpidÿrid,ÿmixedÿabsoluteÿtestvalue(0.999)

Intraclassÿcorrelations
Two-wayÿmixed-effectsÿmodel
Absoluteÿagreement

Randomÿeffects:ÿpidÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿtargetsÿ=ÿÿÿÿÿÿÿÿ50
ÿFixedÿeffects:ÿridÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿratersÿÿ=ÿÿÿÿÿÿÿÿÿ3

--------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿscoÿ|ÿÿÿÿÿÿÿÿICCÿÿÿÿÿÿÿ[95%ÿConf.ÿInterval]
-----------------------+--------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿIndividualÿ|ÿÿÿ.9963082ÿÿÿÿÿÿÿ.9901427ÿÿÿÿ.9982885
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿAverageÿ|ÿÿÿ.9987664ÿÿÿÿÿÿÿ.9966925ÿÿÿÿ.9994289
--------------------------------------------------------------
Fÿtestÿthat
ÿÿICC(1)=1.00:ÿF(49.0,ÿ13.6)ÿ=ÿ0.27ÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿ=ÿ1.000
ÿÿICC(k)=1.00:ÿF(49.0,ÿ13.6)ÿ=ÿ0.81ÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿFÿ=ÿ0.716

.ÿ
.ÿxtregÿscoÿi.rid,ÿi(pid)ÿre

Random-effectsÿGLSÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ150
Groupÿvariable:ÿpidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿÿ50

R-sq:ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿwithinÿÿ=ÿ0.0000ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿÿ3
ÿÿÿÿÿbetweenÿ=ÿ0.0000ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿÿ3.0
ÿÿÿÿÿoverallÿ=ÿ0.0009ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿÿ3

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(2)ÿÿÿÿÿÿ=ÿÿÿÿÿÿ58.40
corr(u_i,ÿX)ÿÿÿ=ÿ0ÿ(assumed)ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿ0.0000

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿscoÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿridÿ|
ÿÿÿÿÿÿÿÿÿÿ2ÿÿ|ÿÿ-.0744089ÿÿÿ.0200366ÿÿÿÿ-3.71ÿÿÿ0.000ÿÿÿÿÿÿ-.11368ÿÿÿ-.0351378
ÿÿÿÿÿÿÿÿÿÿ3ÿÿ|ÿÿ-.1530978ÿÿÿ.0200366ÿÿÿÿ-7.64ÿÿÿ0.000ÿÿÿÿ-.1923689ÿÿÿ-.1138267
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿ_consÿ|ÿÿ-.5092429ÿÿÿÿÿ.29142ÿÿÿÿ-1.75ÿÿÿ0.081ÿÿÿÿ-1.080416ÿÿÿÿ.0619298
-------------+----------------------------------------------------------------
ÿÿÿÿÿsigma_uÿ|ÿÿÿ2.058214
ÿÿÿÿÿsigma_eÿ|ÿÿ.10018325
ÿÿÿÿÿÿÿÿÿrhoÿ|ÿÿ.99763636ÿÿÿ(fractionÿofÿvarianceÿdueÿtoÿu_i)
------------------------------------------------------------------------------

.ÿtestparmÿi.rid

ÿ(ÿ1)ÿÿ2.ridÿ=ÿ0
ÿ(ÿ2)ÿÿ3.ridÿ=ÿ0

ÿÿÿÿÿÿÿÿÿÿÿchi2(ÿÿ2)ÿ=ÿÿÿ58.40
ÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿ=ÿÿÿÿ0.0000

.ÿ
.ÿexit

endÿofÿdo-file


.

The main thing is that you and your colleagues need to decide what you're looking for in terms of "stability", what characteristic is important for your study's interpretability.

[Andrea Burgess]

Thank you for all the feedback. I really appreciate the time you have spent writing back and giving examples. I will re-read your statements and examples, and find those papers.
I do think the xtrans is the unvarnished truth. I had come to the same conclusion that although there is a "significant" change it is small. I will talk to my supervisors about what is best used for interpreting the results regarding the 'stability' of macs.
Kind regards
Andrea.