[Clyde Schechter]

I think this is a good start. There's a bit more I would suggest. Notice that in your first table, for ED presentations the mean in the control phase is 0.25, and in the treatment phase it's 0.23, yet the adjusted mean difference is a whopping 0.12! That looks out of whack. Now, in fact, there's nothing wrong there. The reason is that there is a very strong secular trend in your data. Looking at the regression table, we see that the TimePoint coefficients (which represent the difference in Ed_Presentation from baseline in the untreated condition, get strongly negative at 6 and 9 months. So that secular decrease in ED_Presentation is cancelling out the treatment effect (which is positive). That's why despite a strong treatment effect in the positive direction, the mean ED actually decreases. Explaining that can be difficult, and juxtaposing these things all in the same table highlights the complexity. So I would do it a bit differently. Take the Adjusted Mean DIfference out of that first table. Make another table, 2 x 4, which shows the mean outcome and standard error in each condition at each time period. (The treated-baseline and untreated- 9 months cells will be empty, of course).. That will make it clearer that we have a positive treatment effect that is working in opposition to a negative secular trend and that the two nearly cancel out. After that, then present the adjusted mean differences separately, either in a small table, or just in text.

The coefficient of _cons is just the mean value of ED_Presentation at baseline in the untreated condition (i.e. all the other variables in the model are at 0). You won't actually need that separately because it will appear in the 2 x 4 able I already suggested in the preceding paragraph.

The large residual variance is worth talking about: it is the main reason that your estimates are so imprecise (wide confidence intervals). The variance components at the GP_Cluster and RID levels are not so bad as the residual variance, but even they are still fairly large when you compare them to the average values here. So, I would make a small table showing the variance at each of those levels. I would also add a column showing these in the standard deviation metric (which makes them comparable to the regression coefficients) so that people can see that these variances are of the same order of magnitude as the effects themselves (and in the case of the residual level, considerably larger.) This in turn would justify a recommendation that for better estimates you would need a much less noisy variable or a larger study (or both).

[Elena Mvk]

Thank you Mr Schechter. My analysis is moving at a snail's pace, but it is moving thanks to your advice.

I was wondering if you know what is the best way to approach missign data points.

Cluster	Baseline, n patients	Treatment 1,n	Treatment 2,n	Treatment 3,n
1	30	28	26	25
2	30	29	29	28
3	30	25	25	24

When I generated the On_treatmetn variable, I first assumed that all patients who were at baseline entered the treatment and finished it. But in reality, there were a few patients missing.

Therefore, the On_treatmetn variable was 1 if the patient was actually on treatment.
So, in the case above (table) the total On_treatment phase was 160 = SUm(Cluster1 Treatment) + Cluster 2 (treatment) + Cluster 3 (treatmetn))
not 180 (if all 30 patient in each cluster attended all treatment steps)

How would one indicate the On_treatmetn data for that patient that received the treatment in Step 1 but did not receive it in Steps 2 and 3.
Should they be = . (missing) or = 0 (no treatment) ?


Thank you

[Clyde Schechter]

This is a problem for which there is no good solution. It's a matter of finding the least bad approach.

First of all, we have to distinguish two things: missing vs crossovers. The total number of participants is decreasing across the rows of the table. So people who are not actually in the study any more are truly missing values. Then there is the issue of people who were supposed to get treatment at a particular time, and who are still in the study (getting their outcomes assessed) but for whatever reason did not actually get treatment. These are crossovers.

Both of them create severe challenges to unbiased data analysis. But in different ways and there are different conventions about how to handle them.

Let's do crossovers first. Your context appears to be health care, which is my area as well. The convention is clearly established in health care research that the primary analysis of data with crossovers is the "intention to treat (ITT)" analysis. In the ITT analysis, everything is analyzed as if the originally assigned treatment had been followed. So all of the people in your study who failed to receive treatment at a time when they should have, should still have on_treatment coded as 1 for those time periods. There is only one exception to this convention: if you are doing a safety trial of a new intervention, then you analyze them with the treatment status they actually received (as treated analysis). Now, that is the primary analysis, but one often presents secondary analyses as well. In the usual situation, where the ITT analysis is primary, the as treated analysis might be presented as secondary. If you are lucky, the results will be nearly the same, which implies that the non-adherence to treatment assignment didn't make that much difference. If you are unlucky the results will be markedly different and it will be impossible to draw any clear conclusions, though people will lean towards accepting the ITT analysis.

Turning to missing data (dropouts or other reasons ). In your analysis things are fairly straightforward as you have only time and on_treatment as predictors, along with GP_cluster and RID level effects. The nice thing is that none of those variable can ever have a missing value because you assign them! So the only way you can have a missing observation is if they did not have an outcome measurement at some time(s) during the study. That makes life simpler because you do not even have to think about whether it would make sense to do multiple imputation: when only the outcome is missing, multiple imputation does not accomplish anything. And your life is simpler because multiple imputation is a lot of work for both you and your computer. The other lucky situation here is that you are using -mixed- for your analysis, which uses maximum likelihood estimation. So if your missing outcomes are missing at random (which is a technical term that does not mean what most people think it means) your estimates based only on complete cases will be unbiased anyway. If your missing outcomes are missing not at random, then there is no really good alternative. You present an analysis with complete cases, and supplement that with sensitivity analyses (sometimes called robustness analyses). In these analyses, you can do things like best-case and worst-case scenarios for imputing the missing values and running analyses with those imputations. For example, to take an extreme worst case scenario, if your hypothesis is that the intervention increases the outcome, you could replace all the missing values by the lowest possible outcome value when on_treatment = 1 and by the highest possible outcome value when on_treatment = 0. That analysis stacks the deck against your hypothesis in the strongest possible way--so if your hypothesis still holds up, it's robust. You might also want to do a best case scenario if you want to set an upper bound on just how much your treatment might increase the outcome. That one would work the opposite: in the on_treatment condition you replace missing outcomes by maximum possible outcomes, and by minimum possible outcomes when on_treatment = 0. Other less extreme analyses are often done--especially since the extreme worst case scenario usually does defeat your hypothesis! So you might do things less extreme like, look at the non-missing outcome values for a person with a missing outcome and replace the missing with the lowest outcome ever observed for that particular person during the treatment assignment period, and by their highest outcome ever observed during the off-treatment assignment period. This is sometimes called worst case carried forward. And you can similarly do a non-extreme favorable analysis with best case carried forward. Finally there are analyses that don't explicitly seek to stack the deck one way or another: replace each missing outcome value for a given person by the last preceding non-missing outcome value.

So those are various approaches. to these problems. As I said at the outset, none of them is truly satisfactory. None of them can be relied upon to give you unbiased estimates (except for the complete cases analysis with maximum likelihood estimation when the missing values are, in the technical sense, missing at random.) You may be asking yourself whether you should try to check whether the missing values are missing at random. Which sounds reasonable, except that there is a catch. If you correctly understand the term missing at random, you will realize that it is literally impossible to do any such test within your data: it is an inherently untestable assumption. The only evidence that can be brought to bear on the issue would be external information about the process that drove the missingness process itself--and such information is usually not available.

If you want an overview of the awful world of missing data analysis, take a look at https://statisticalhorizons.com/wp-c...aterials-1.pdf.

[Elena Mvk]

Thank you Mr Schechter.

I get the following composition of On_Treatment.

table Step_n TimePoint, c(min On_Trt max On_Trt )

----------------------------------
Step of   |
transitio |
n into    |  0 = baseline, 3=at 3 
intervent |months, 6= 6 month, 9 =
ion       |        9 months       
(1,2,3)   |    0     3     6     9
----------+-----------------------
        1 |    0     1     1     1
          |    0     1     1     1
          | 
        2 |    0     0     0     0
          |    0     0     1     1
          | 
        3 |    0     0     0     0
          |    0     0     0     1
----------------------------------

In Step 2 - there were 30 patients randomised, but only 28 commenced treatment in TimePoint 6 (mths).
Following the ITT approach, should these randomised patients, who did not receive treatment, still be included in the analysis?

[Clyde Schechter]

If their outcomes were measured, then, yes, they should be included in the analysis and they should be analyzed as if they were, in fact, on treatment. (If their outcomes were not measured at that time, then there is no way to include them in the primary analysis at all.)

[Renira Corinne Angeles]

This thread has been really useful for me whilw analyzing the results of a stepped wedge cluster RCT. I do have follow-up questions.

I have similar datastructure as Elena: timepoints: 0, 6, 12, 18 and 24 month, municipality (n=3), individual ID, and randomization group (n=3).

Do I include outcomes from 24 month when creating the "has_begun_treatment" variable as demonstrated earlier in this thread. See table below. Or do I analyze the 24 month follow-up in a separate analysis? If so, do anyone have any hints or advice on how?

version 16: table randomization month, c(mean treated)

------------------------------------------
wave of     |
randomizati |            month            
on          |    0     6    12    18    24
------------+-----------------------------
 First wave |    0     1     1     1     1
Second wave |    0     0     1     1     1
 Third wave |    0     0     0     1     1
------------------------------------------

[Clyde Schechter]

Include the 24 month data when creating has_begun_treatment. There is no reason to do a separate analysis on that data, unless there is some specific research goal that singles out that time period for special attention. (For example, if the intervention itself is limited in duration, there might be a separate research goal to see if the effects persist between months 18 and 24. That would be a separate analysis.)

[Renira Corinne Angeles]

Many thanks for your reply and input, Clyde!

Actually, I didn't make it clear that the 24 month data collection is to investigate exactly what you write, if the effect still persist. So,I understand it has to be a separate analysis then - and has_begun_treatment should only include up to month 18?

[Clyde Schechter]

Well, if the only analyses to be done with the 24 month data is to determine whether the effect persists, then you should code3 has_begun_treatment as missing value in month 24. That way, when you do the regular analysis, observations at 24 months will be omitted. Then you can later do a separate analysis, which will not involve the has_begun_treatment variable, involving only months 18 and 24.

[Renira Corinne Angeles]

Thanks for the suggestions.

If I understand correctly, you mean that I add month 18 and 24 as dummy variables, like this

mixed y x1 x2 xn i.municipality month18 month24  || individual level id:

or, do the regression while only adding the data within month 18 and 24?

I am not sure how to interpret the follow up analysis. Would it just be as simple as interpreting the effect of the time dummy for month 24? Or should I transform Y to a difference between month 18 and 24?

Again, thanks for your input, professor Clyde.

[Clyde Schechter]

I assume your speaking in #25 of how you would assess persistence of effect between months 18 and 24.

First, I wouldn't use separate indicators ("dummies") for any of the months, not in this analysis, nor in the original analysis of the effect of the intervention up to month 18. Better is to have a single variable, month, which takes on the values 0, 3, 6, 12, 18, and 24. Then for the persistence analysis:

mixed y x1 x2 xn i.municipality i.month if inlist(month, 18, 24) || individual_level_id:

If you are not familiar with factor-variable notation, as used here, it is time to read -help fvvarlist-.

You would estimate the change between months 18 and 24 as the coefficient of the 24.month variable.

[Renira Corinne Angeles]

This was very helpful. Thanks a lot!!

[Renira Corinne Angeles]

Hi
I'm reponening this post from June 2022. I would need some guidance in the interpretation of the results. I'm estimating the treatment effect in a stepped-wedge trial over a two year period with a treatment interval of 6 months (t0 - basline, t1 - month 6, t2 - month 12, t3 - month 18, t4 - month 24). See output and othe information below. The Y is "rsstotal". The variable "treat" displays the treatment effect, or more correctly: the average difference after treatment between intervention (condition) and control (condition) group. How do I collect the intervention effect for the control group? I understand the betacoefficient for treat (.4903281) is the (insignificant) difference between intervention-control. But how should I articulate/inform the effect in the control group (in the text)? Is it ofr instance correct to write that there is an (insignificant) increase in the intervention group, and no effect for the control group? ALternatively, I could use the constant which basically says the baseline value for the control group, but when I compare the values of rsstotal (the Y) between sum and the mixed effect table below, the summarize value show 15.9 and not 20.
I hope this was readable.

mixed rsstotal rudiadldays  pwdage_real cgage_real pwd_female cg_female spouse careg_con  padltot_ i.municipality i.month i.treat  || newid:

Performing EM optimization ...

Performing gradient-based optimization: 
Iteration 0:   log likelihood = -2246.0756  
Iteration 1:   log likelihood = -2246.0756  

Computing standard errors ...

Mixed-effects ML regression                     Number of obs     =        654
Group variable: newid                           Number of groups  =        268
                                                Obs per group:
                                                              min =          1
                                                              avg =        2.4
                                                              max =          5
                                                Wald chi2(15)     =     128.38
Log likelihood = -2246.0756                     Prob > chi2       =     0.0000

------------------------------------------------------------------------------
    rsstotal | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
rudiadldays_ |   .1071154   .0330111     3.24   0.001     .0424148    .1718159
 pwdage_real |  -.0508273   .0428743    -1.19   0.236    -.1348595    .0332049
  cgage_real |  -.1600696   .0618367    -2.59   0.010    -.2812672   -.0388719
  pwd_female |  -1.052238   1.256061    -0.84   0.402    -3.514072    1.409597
   cg_female |    3.27658   1.107287     2.96   0.003     1.106338    5.446822
      spouse |   6.673438   1.807912     3.69   0.000     3.129995    10.21688
   careg_con |    2.00993   .7772371     2.59   0.010     .4865735    3.533287
    padltot_ |   .4531269   .1059743     4.28   0.000     .2454212    .6608326
             |
municipality |
     Bergen  |  -.6347728   1.255292    -0.51   0.613    -3.095099    1.825554
     Baerum  |  -1.330146   1.332206    -1.00   0.318    -3.941221    1.280929
             |
       month |
          6  |  -.7901302   .6344234    -1.25   0.213    -2.033577    .4533169
         12  |  -2.161259   .8586724    -2.52   0.012    -3.844226   -.4782915
         18  |  -1.737958   1.387732    -1.25   0.210    -4.457862     .981946
         24  |   .0284752   1.358804     0.02   0.983    -2.634731    2.691681
             |
     1.treat |   .4903281   .8454851     0.58   0.562    -1.166792    2.147448
       _cons |   20.16042   4.796864     4.20   0.000     10.75874     29.5621
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects parameters  |   Estimate   Std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
newid: Identity              |
                  var(_cons) |   43.88024   5.366791      34.52729    55.76678
-----------------------------+------------------------------------------------
               var(Residual) |   31.40953   2.296634      27.21588    36.24937
------------------------------------------------------------------------------
LR test vs. linear model: chibar2(01) = 174.53        Prob >= chibar2 = 0.0000

This is treat:

version 16: table randomization month, c(mean treat)

------------------------------------------
wave of     |
randomizati |            month            
on          |    0     6    12    18    24
------------+-----------------------------
 First wave |    0     1     1     1     0
Second wave |    0     0     1     1     0
 Third wave |    0     0     0     1     0
------------------------------------------