How to perform hypothesis test for effect modification in subgroup analyses after -teffects- commands?

[Philip Jones]

I am using the -teffects ipwra- command to perform analysis on an observational dataset. The command is straightforward and very useful. However, part of my project is investigating whether any effect modification exists for subgroups of the main dataset. For example, whether there is an effect modification of sex on the primary outcome, and whether this effect is statistically significant.

Normally, I would do this by incorporating an interaction term in a logistic regression model (for a binary outcome variable). However, using the -teffects- commands, it is not clear to me how I would investigate this.

Here is an example, where primary and sex are binary indicator outcome variables, group is a binary treatment group variable, and age is a continuous variable:

I start by investigating the overall treatment effect, after controlling for sex and age (both sex and age are potentially predictors of both treatment and outcome):

 teffects ipwra (primary i.sex c.age, logit) (group i.sex c.age, logit), ate

Iteration 0:   EE criterion =  5.391e-16  
Iteration 1:   EE criterion =  2.137e-29  

Treatment-effects estimation                    Number of obs     =        464
Estimator      : IPW regression adjustment
Outcome model  : logit
Treatment model: logit
----------------------------------------------------------------------------------------------
                             |               Robust
                     primary |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------------------+----------------------------------------------------------------
ATE                          |
                       group |
(sevoflurane vs isoflurane)  |  -.0546617   .0408894    -1.34   0.181    -.1348034    .0254801
-----------------------------+----------------------------------------------------------------
POmean                       |
                       group |
                 isoflurane  |   .3005392   .0297764    10.09   0.000     .2421786    .3588999
----------------------------------------------------------------------------------------------

I then investigate the treatment effects in each of the subgroups of sex, first for females:

. teffects ipwra (primary c.age, logit) (group c.age, logit) if sex==1, ate

Iteration 0:   EE criterion =  7.012e-17  
Iteration 1:   EE criterion =  1.907e-32  

Treatment-effects estimation                    Number of obs     =         97
Estimator      : IPW regression adjustment
Outcome model  : logit
Treatment model: logit
----------------------------------------------------------------------------------------------
                             |               Robust
                     primary |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------------------+----------------------------------------------------------------
ATE                          |
                       group |
(sevoflurane vs isoflurane)  |  -.1590846   .0898707    -1.77   0.077     -.335228    .0170589
-----------------------------+----------------------------------------------------------------
POmean                       |
                       group |
                 isoflurane  |   .3737519   .0677898     5.51   0.000     .2408862    .5066175
----------------------------------------------------------------------------------------------

and then for males:

. teffects ipwra (primary c.age, logit) (group c.age, logit) if sex==2, ate

Iteration 0:   EE criterion =  9.539e-25  
Iteration 1:   EE criterion =  6.300e-33  

Treatment-effects estimation                    Number of obs     =        367
Estimator      : IPW regression adjustment
Outcome model  : logit
Treatment model: logit
----------------------------------------------------------------------------------------------
                             |               Robust
                     primary |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------------------+----------------------------------------------------------------
ATE                          |
                       group |
(sevoflurane vs isoflurane)  |  -.0268644   .0457452    -0.59   0.557    -.1165234    .0627947
-----------------------------+----------------------------------------------------------------
POmean                       |
                       group |
                 isoflurane  |   .2807117   .0329393     8.52   0.000     .2161519    .3452715
----------------------------------------------------------------------------------------------

This gives me the estimated treatment effect (and 95% CIs) for each subgroup of sex.

My question is, how do I investigate whether or not the difference between sexes is statistically significant?

Many thanks for any help with this.

Phil

[Philip Jones]

I am not sure if this subgroup analysis can be directly done within the -teffects- series of commands (though this is a shame since the whole point of these commands is to simplify things).

So I think I may have to do this manually. To share with the group, I offer the following:

I create a new propensity score (i.e., a logit model predicting the receipt of treatment) manually containing all of my covariates except for the covariate for the subgroup of sex (i.e. I am investigating the effect of sex, so I will eliminate sex from the propensity score model so I am not controlling for sex while investigating the subgroup effect of it at the same time).

logit group c.age
predict ps

replace ipw = 1/ps if group==1
replace ipw = 1/(1-ps) if group==0

I then perform weighted logistic regression on my outcomes of interest where the independent covariates are the treatment group the patient is in, the subgroup variable, and their interaction. This gives me the needed P value for interaction. As follows:

logistic primary i.group##i.sex [pw=ipw]

Logistic regression                             Number of obs     =        464
                                                Wald chi2(3)      =       3.66
                                                Prob > chi2       =     0.3007
Log pseudolikelihood =   -541.231               Pseudo R2         =     0.0066

-----------------------------------------------------------------------------------
                  |               Robust
          primary | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
            group |
     sevoflurane  |   .4637321   .2135961    -1.67   0.095      .188019    1.143754
                  |
              sex |
            Male  |   .6582955   .2196998    -1.25   0.210     .3422476    1.266197
                  |
        group#sex |
sevoflurane#Male  |   1.871419   .9688985     1.21   0.226     .6783824    5.162589
                  |
            _cons |   .5950657   .1725463    -1.79   0.073     .3370925    1.050463
-----------------------------------------------------------------------------------

I then attempt to replicate the above using the -teffects ipw- package:

. * subgroup analysis using teffects package
. teffects ipw (primary) (group c.age, logit) if sex==1, pom // female

Treatment-effects estimation                    Number of obs     =         97
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: logit
------------------------------------------------------------------------------
             |               Robust
     primary |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
POmeans      |
       group |
 isoflurane  |   .3738608   .0677752     5.52   0.000     .2410238    .5066978
sevoflurane  |   .2140548   .0595845     3.59   0.000     .0972714    .3308383
------------------------------------------------------------------------------


. * subgroup analysis using logistic regression
. qui logistic primary i.group [pw=ipw] if sex==1

. margins i.group

Adjusted predictions                            Number of obs     =         97
Model VCE    : Robust

Expression   : Pr(primary), predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       group |
 isoflurane  |   .3730666   .0680974     5.48   0.000     .2395982     .506535
sevoflurane  |   .2162709   .0609089     3.55   0.000     .0968916    .3356502
------------------------------------------------------------------------------

 
. teffects ipw (primary) (group c.age, logit) if sex==2, pom // male

Treatment-effects estimation                    Number of obs     =        367
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: logit
------------------------------------------------------------------------------
             |               Robust
     primary |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
POmeans      |
       group |
 isoflurane  |   .2810828   .0329973     8.52   0.000     .2164093    .3457563
sevoflurane  |   .2538208   .0319609     7.94   0.000     .1911786    .3164629
------------------------------------------------------------------------------


. * subgroup analysis using logistic regression
. qui logistic primary i.group [pw=ipw] if sex==2

. margins i.group

Adjusted predictions                            Number of obs     =        367
Model VCE    : Robust

Expression   : Pr(primary), predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       group |
 isoflurane  |   .2814693   .0334288     8.42   0.000     .2159501    .3469886
sevoflurane  |   .2537074   .0320237     7.92   0.000     .1909421    .3164727
------------------------------------------------------------------------------

As you can, the expected frequencies between the two methods are close, but not exactly equal. I suspect this has something to do with the estimation in the -teffects- package being based on GMM?

Thanks for any feedback regarding my technique or the ability to actually do subgorup analyses directly within the -teffects- packages.

Phil

[Christopher Barrie]

I similarly would be interested in an answer to this question. It strikes me, and I'd be happy for the community to correct me if I'm wrong, that teffects would be a lot more useful if it enabled users easily to generate matched datasets (as can be achieved, for example, in the MatchIt package in R). This would mean users could easily perform subgroup analyses on the matched (or weighted) data using conventional parametric methods. The disadvantage of the set of commands provided by teffects seems to be precisely that each step in this procedure is wrapped up in one command, thus limiting the range of analyses that can be performed. A workaround is offered here to see matched obs. but is rather clunky, and a column of identifiers can be generated with the postestimation generate option. Given that matched data is being generated in the background, though, surely it wouldn't be difficult to provide functionality to generate a new matched dataset?

[shem shen]

I am also interested in this question. Can we really use IPW models to study heterogeneous treatment effects? I only know that we can use it to study ATT, ATE, ATC. I find it problematic to exclude sex from the propensity score model because you want to examine heterogeneous treatment effect across gender. In doing so, you estimated a misspecified propensity score model. I would prefer to first of all estimate the correct propensity score model, and then estimate the IPW model, and finally add the interaction term between sex and the treatment variable into the IPW model, but I do not know if it is statistically correct to do so.