Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Mixed model syntax and output question

    Hello --

    I am trying to fit a mixed effect linear model to some data. For context, I have 4 raters (raterid) who each rate 3 notes (noteid) from 10 different learners (residentID). Each learner is rated by the same 4 raters, providing a final score (tscore). Hence, noteid is nested in raters, who are nested in learners.

    Data structure:
    noteID residentID Tscore reviewer_n
    1041901 8 11 1
    1041901 8 9 2
    1041901 8 12 3
    1041901 8 11 4
    1041902 8 12 1
    1041902 8 12 2
    1041902 8 12 3
    1041902 8 9 4
    1041903 8 11 1
    1041903 8 11 2
    1041903 8 12 3
    1041903 8 10 4
    1111901 10 10 1
    1111901 10 10 2
    1111901 10 11 3
    1111901 10 6 4
    1111902 10 9 1
    1111902 10 8 2
    1111902 10 10 3
    1111902 10 8 4
    1111903 10 11 1
    1111903 10 10 2
    1111903 10 11 3
    1111903 10 6 4

    For this project, I would like to estimate inter-rater reliability and examine rater differences.
    1. To estimate Inter-rater reliability and obtain the ICC I used the following syntax:

    mixed tscore || residentID: || noteID:
    estat icc
    Relevant output for Intraclass correlation:

    ------------------------------------------------------------------------------
    Level | ICC Std. Err. [95% Conf. Interval]
    -----------------------------+------------------------------------------------
    residentID | .1368313 .0841699 .0377377 .3905274
    noteID|residentID | .1368313 .0841699 .0377377 .3905274
    ------------------------------------------------------------------------------

    Is it correct to assume that for residentID, the proportion variance in tscore between residents is .14? Also, why am I getting identical values for the other nested level of the model (noteID|residentID)? Is my nesting incorrect?
    1. If I want to estimate rater differences, would I use the following syntax?
    mixed score i.raterid || residentID: ||noteID:

    Because rater is a categorical variable, it provides a reference group. Is it possible to estimate an overall omnibus effect of rater?


    Thanks!

  • #2
    Please use code delimiters plut - dataex - for sharing data/command/output, as recommended in the FAQ.

    That being said, the example in #1 presents just 1 resident per noteid, hence the same results for the ICC.

    To end, I wonder whether a mixed model is the best match with ratings.

    Hopefully you'll get some theoretical background on this as well.
    Best regards,

    Marcos

    Comment


    • #3
      Originally posted by cjp moons View Post
      noteid is nested in raters, who are nested in learners.
      From your description as well as from the very brief listing, it looks more as if note is nested under learner, and the latter crossed with rater.

      Maybe try something like the following to get at the intraclass correlation for rater, assuming the pattern for the note variable is similar in the data that you don't show, and that it is numeric.
      Code:
      generate byte note = real(substr(string(noteID), -1, 1))
      mixed Tscore i.note || _all: R.residentID || reviewer_n:

      Comment


      • #4
        I apologize for the posting error. I will take note for next time. Thank you for your thoughts

        Comment


        • #5
          My apologies. I have included some sample data below.

          Code:
          * Example generated by -dataex-. To install: ssc install dataex
          clear
          input long noteID byte(residentID tscore) long reviewer_n
           1  8 11 4
           1  8  9 1
           1  8 12 2
           1  8 11 3
           2  8 12 4
           2  8 12 1
           2  8 12 2
           2  8  9 3
           3  8 11 4
           3  8 11 1
           3  8 12 2
           3  8 10 3
           4 10 10 4
           4 10 10 1
           4 10 11 2
           4 10  6 3
           5 10  9 4
           5 10  8 1
           5 10 10 2
           5 10  8 3
           6 10 11 4
           6 10 10 1
           6 10 11 2
           6 10  6 3
           7  9  9 4
           7  9 11 1
           7  9 11 2
           7  9  7 3
           8  9 10 4
           8  9 11 1
           8  9 11 2
           8  9  6 3
           9  9  8 4
           9  9  8 1
           9  9 12 2
           9  9  8 3
          end
          label values reviewer_n reviewer_n
          label def reviewer_n 1 "rater1", modify
          label def reviewer_n 2 "rater2", modify
          label def reviewer_n 3 "rater3", modify
          label def reviewer_n 4 "rater4", modify
          To clarify the design, all 4 raters reviewed each resident note (30 total notes). Each resident had 3 of their notes reviewed 4 times - once by each rater, for a total of 12 note review observations per resident.

          I would like to estimate ICCs for the data, but accounting for the inherent nesting structure of multiple observations for notes and residents.

          Does this help to clarify the appropriate syntax that I would use?

          Comment


          • #6
            Originally posted by cjp moons View Post
            Does this help to clarify the appropriate syntax that I would use?
            What are the "notes"? Are they just repetitions (repeated measurements) of an exercise by the "learners" (whom you also refer to as "residents")? If so then, what I showed above might be appropriate.

            On the other hand, from the nature of the identifier (noteID) that you showed in your initial post, a "note" might be of a patient encounter (say a physical examination) by a resident (that is physician-in-training). If so, then it might be possible to consider note (in the manner that it represents patients with a similar medical history and presentation) as another crossed random effect.

            I think what you need to do is sit down and write out the so-called data-generating process by which the "tscores" come about. Then you can write the statistical model (in the language of mixed's syntax) to represent that process. From fitting that model to the data, you can estimate the magnitudes of the various sources of random variation (the variance components) and compute the intraclass correlation coefficient.

            Comment


            • #7
              Joseph --

              Your interpretation is correct ...

              The scores (tnotes) are repetitions (repeated measurements) of an exercise by "learners". Indeed, a note refers to a patient encounter by a resident (physician-in-training), however the notes and patients will vary (not similar medical histories or conditions).

              I'm struggling to write syntax given limited mixed modelling training. Can you recommend a starting reference?

              In terms of your recommended code:

              Code:
               
               generate byte note = real(substr(string(noteID), -1, 1)) mixed Tscore i.note || _all: R.residentID || reviewer_n:
              I'm not clear why you suggested modeling note as a categorical IV. Can you clarify?

              Also, how can one estimate the ICC from the output results?

              Thanks so much for your help. I am aware that I am asking a lot

              Chris

              Comment

              Working...
              X