Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Randomizatio inference (beyond "permute")

    Hi,

    I was wondering if there are any tools for randomization inference / Fisher permutation tests, that go beyond what can be achieved by the permute command. I wrote something myself a while back and was wondering whether it's worth making that into a package and putting it up somewhere.

    Simple --- and probably common --- example of an analysis, which can't be done using permute: (Please correct me if I'm wrong, then my post is pointless)
    An an RCT classes within school are randomly assigned to treatment. Either the whole class or no one in a class gets treatment, but treatment is stratified by schools so that within each school the same number of classes are in treatment and control group respectively. Now it is desired to use simple permutation techniques to assess whether the difference in test scores between treated and non-treated students is significant (possibly controlling for some student-level covariates). The treatment permutation would've to account for the clustered treatment assignment, as well as the strata.

    Best,

    Simon

  • #2
    I programmed something that I might soon offer to other's as a package. Before doing so I just wanted to ask one more time whether there is nothing like this available here or somewhere else?

    Comment


    • #3
      I missed the earlier post, sorry - however, I am confused: permute can be used with any Stata command that follows standard Stata syntax - what are you trying to permute that does not follow standard syntax? maybe I don't understand your first post but I would imagine that the analysis you want for that is fairly standard; maybe if you spelled out what the analysis would look like someone could give a better answer

      Comment


      • #4
        Assume I have 10 classes of 20 students each. I observe test-scores after I randomly assigned class-level treatment (e.g. additional materials). Now I want to see whether the students in treatment classes are better than the students in non-treatment classes, but have to take into account that treatment assignment was by class.

        Code:
        reg outcome treatment, cluster(class)
        this would be the standard way of doing it.

        Now instead I want to use a permutation test. And while permute knows
        Code:
        strata(varlist)
        , it doesn't seem to be able to re-create groupwise treatment assignments, which are fairly standard. If I neglect this in the permutation test, it will likely be overrejecting, as it ignores the within cluster correlation induced my my method of randomization.

        Am I missing something?
        Last edited by Simon Heß; 06 Jun 2016, 11:50.

        Comment


        • #5
          Simon:
          Example #2 under -permute- entry in Stata .pdf manual might be helpful.
          Kind regards,
          Carlo
          (Stata 19.0)

          Comment


          • #6
            Thanks Carlo, I assume you mean "Example 2: Permutation tests with ANOVA".

            While I see that it mentions "a randomized complete-block design" this doesn't seem to be accounted for in the actual analysis, as permute is just called as:
            Code:
            permute treatment treatmentF=r(F) modelF=e(F), reps(1000) strata(subject) saving(permanova) nodots: panova y treatment subject
            (where panova is a wrapper that simply returns the F-stat from the anova)

            Is what I am asking maybe still unclear? If so, could you clarify what you don't understand about it?

            Another try to clarify:
            If the original randomization was not on the individual level, tests based on permute are not strictly re-randomization tests. To calculate the distribution of a test statistic based on randomization inference, I must replicate the original randomization process. Permute reshuffles treatment at the observation level, which does not depict the orignal randomization process given clustered/blocked treatment assignment.

            Hence my question: Is there a way to make permute account for this? If not, I might wrapup my code into a package and share it, given there's an interest for that.
            Last edited by Simon Heß; 07 Jun 2016, 09:20.

            Comment


            • #7
              Simon:
              thaks for providing further details.
              My intent was to point you out to the example (that you rightly spotted) seemingly nearer to your research goal.
              Actually, it was not that helpful.
              In the past I was very fond of resampling methods, but in your case I would ask for nothing more than -regress-.
              I do not think that, by default, -permute- can account what you're after.
              Kind regards,
              Carlo
              (Stata 19.0)

              Comment


              • #8
                Originally posted by Carlo Lazzaro View Post
                In the past I was very fond of resampling methods, but in your case I would ask for nothing more than -regress-.
                Not sure I fully understand your statement here. The problem with test based on OLS and normal or Sandwhich-type standard errors could be the small number of clusters. Anyhow tests based on data from clustered treatment assignment trials are not a rarity I believe

                Comment


                • #9
                  Simon:
                  you're right; clustered treatment assignment trials are well covered in literature.
                  And you're right again that the problems related to clustered standard errors (SEs) may rest on the number of clusters.
                  I simply meant that, in your case, I would stick with OLS with clustered SEs and forget permutation.
                  Kind regards,
                  Carlo
                  (Stata 19.0)

                  Comment


                  • #10
                    Thanks for Clarifying Carlo, my "case" was a fictitious one, any my question whether this would be possible in stata. I'll take the responses here as a "no", but also also as indication that such a functionality wouldn't be in high demand

                    Comment


                    • #11
                      Permute is an elegant command, and it seems to me that the option to cluster the permutations would be a welcome addition to the option list, alongside the already existing strata option. I agree that clustered design is not a reason to drop permutation tests, but a permute command that ignores clustering likely over-rejects for the same reasons an unadjusted regression would. So I think the feature would be in demand (at least I'd use it with delight). Meanwhile I'll try & work on an ugly workaround.

                      Comment


                      • #12
                        I already wrote this addition to the permute command, plus a few others. Send me an email and I'll share it with you

                        Comment


                        • #13
                          Alongside the package (-ritest-) I wrote a short Stata Journal article explaining how to use the command to perform various variants of randomization inference in Stata. The article is published here: http://www.stata-journal.com/article...article=st0489

                          With the ritest-command it becomes very easy to perform randomization infernce for any kind of command in Stata. -ritest- is flexible enough to work with arbitratily complex randomiztaion methods. In particular it is very easy to deal with clustered randomization, pairwise randomization, or user-written any randomization routines in Stata.

                          Installation
                          As always with the Stata Journal the package can be installed through:
                          Code:
                          net sj 17-3
                          I will try to provide regular updates and bugfixes, so I recommend installing the package through my github inseead, which is easy:
                          Code:
                          net describe ritest, from(https://raw.githubusercontent.com/simonheb/ritest/master/)
                          Usage
                          The syntax of the command is close to what that of Stata’s permute. For example, to conduct randomization inference after clustered & stratified treatment assignment, using the t-statistic of an ATE regression as the evaluation statistic run:
                          Code:
                          ritest treatment _b[treatment]/_se[treatment], cluster(class_id) strata(school_id): reg y treatment x
                          (This just permutes treatment across classes within strata and computes the t-statistic of the ATE for each permutation)

                          Or if, instead of permuting, one wants to use the original randomization script to perform the re-randomizations one, but this time use the coefficient estimate as the evaluation statistic, simply write:
                          Code:
                          ritest treatment _b[treatment], samplingprogram(assignment_programm) samplingprogramoptions("school(school_id) class(class_id)"):  reg y treatment x
                          (This repeatedly calls a program the user has to define beforehand: assignment_programm, school(schoold_id) class(class_id))

                          I believe this makes it much easier to do such things with Stata. Would be happy to hear what you think (via email).

                          Comment

                          Working...
                          X