Test for differences and Immediate Impact from XTITSA

[Andrew Adkins Second]

Hello,

I'm attempting to identify average volume pre/post intervention, test differences, then perform an ITSA where I examine changes after the policy went into effect. I'm then asked to examine the immediate monthly change (_x2022m7) but instead as the immediate change in 2022m7 vs. the average predicted value in the pre-period or a specified month in the pre-period as opposed to the intercept. Is this feasible potentially using a postestimation command such as lincom _b[_x2022m7]-something?

Some questions I had:

1. Should I also be incorporating an i.period and/or i.monthly variable into the model?
2. How do I re-estimate an immediate impact (_x2022m7) vs. the values at another specified point (e.g. 2022m5) or vs. the entire pre-period as opposed to the intercept?

input double TRX_per10000 byte period float(month year monthly state_num)
1.0727272727272728 1  1 2022 744  1
1.0545454545454547 1  2 2022 745  1
.9727272727272727 1  3 2022 746  1
.9727272727272727 1  4 2022 747  1
1.2636363636363637 1  5 2022 748  1
1.4181818181818182 2  7 2022 750  1
1.3363636363636364 2  8 2022 751  1
.9090909090909091 2  9 2022 752  1
.5636363636363636 2 10 2022 753  1
.7545454545454545 2 11 2022 754  1
.7181818181818181 2 12 2022 755  1
10.882352941176471 1  1 2022 744  2
10.823529411764705 1  2 2022 745  2
12.117647058823529 1  3 2022 746  2
11.117647058823529 1  4 2022 747  2
13.882352941176471 1  5 2022 748  2
19 2  7 2022 750  2
19.705882352941178 2  8 2022 751  2
17.647058823529413 2  9 2022 752  2
15.411764705882353 2 10 2022 753  2
18.58823529411765 2 11 2022 754  2
28 2 12 2022 755  2
1.75625 1  1 2022 744  3
1.5374999999999999 1  2 2022 745  3
1.5374999999999999 1  3 2022 746  3
1.4 1  4 2022 747  3
1.61875 1  5 2022 748  3
2.3000000000000003 2  7 2022 750  3
1.9249999999999998 2  8 2022 751  3
1.2625 2  9 2022 752  3
1.1125 2 10 2022 753  3
1.0562500000000001 2 11 2022 754  3
.81875 2 12 2022 755  3
5.402985074626866 1  1 2022 744  4
4.134328358208956 1  2 2022 745  4
2.985074626865672 1  3 2022 746  4
2.164179104477612 1  4 2022 747  4
3.074626865671642 1  5 2022 748  4
6.313432835820896 2  7 2022 750  4
5.029850746268656 2  8 2022 751  4
6.074626865671641 2  9 2022 752  4
9.208955223880597 2 10 2022 753  4
8.432835820895523 2 11 2022 754  4
7.447761194029851 2 12 2022 755  4
4.447826086956522 1  1 2022 744  5
5.333695652173914 1  2 2022 745  5
5.573913043478261 1  3 2022 746  5
5.091304347826087 1  4 2022 747  5
5.4206521739130435 1  5 2022 748  5
5.839130434782608 2  7 2022 750  5
5.935869565217391 2  8 2022 751  5
5.417391304347826 2  9 2022 752  5
5.4543478260869565 2 10 2022 753  5
5.206521739130435 2 11 2022 754  5
5.357608695652174 2 12 2022 755  5
6.057142857142857 1  1 2022 744  6
5.414285714285714 1  2 2022 745  6
5.892857142857143 1  3 2022 746  6
5.478571428571429 1  4 2022 747  6
4.621428571428572 1  5 2022 748  6
8.064285714285715 2  7 2022 750  6
6.628571428571429 2  8 2022 751  6
4.75 2  9 2022 752  6
4.642857142857143 2 10 2022 753  6
4.171428571428572 2 11 2022 754  6
4.535714285714286 2 12 2022 755  6
3.7974683544303796 1  1 2022 744  7
4.278481012658228 1  2 2022 745  7
3.4430379746835444 1  3 2022 746  7
4.063291139240506 1  4 2022 747  7
3.7721518987341773 1  5 2022 748  7
4.088607594936709 2  7 2022 750  7
4.443037974683544 2  8 2022 751  7
3.20253164556962 2  9 2022 752  7
3.7341772151898733 2 10 2022 753  7
3.9873417721518987 2 11 2022 754  7
3.569620253164557 2 12 2022 755  7
8.363636363636363 1  1 2022 744  8
6.2272727272727275 1  2 2022 745  8
4 1  3 2022 746  8
4.590909090909091 1  4 2022 747  8
6.409090909090908 1  5 2022 748  8
12.636363636363637 2  7 2022 750  8
10.045454545454545 2  8 2022 751  8
11.999999999999998 2  9 2022 752  8
14.499999999999998 2 10 2022 753  8
14.454545454545455 2 11 2022 754  8
14.090909090909092 2 12 2022 755  8
5.1000000000000005 1  1 2022 744  9
4.75 1  2 2022 745  9
4.75 1  3 2022 746  9
4.4 1  4 2022 747  9
5.15 1  5 2022 748  9
13.4 2  7 2022 750  9
9.9 2  8 2022 751  9
9.200000000000001 2  9 2022 752  9
10.25 2 10 2022 753  9
9.35 2 11 2022 754  9
6.7 2 12 2022 755  9
1.9108695652173913 1  1 2022 744 10
end
format %tm monthly

My code is as follows:

* Declare data to be time-series data
xtset state_num monthly

* sort by ID and time
sort state_num monthly

di monthly("2022m7","YM")

/**************************
AVERAGE VALUES PRE/POST
**************************/
ttest TRX_per10000, by(period)

/*************
ITSA ANALYSIS
*************/
*Overall
xtitsa TRX_per10000, single trperiod(2022m7) vce(robust) posttrend fig replace

My output is as follows:

. xtitsa TRX_per10000, single trperiod(2022m7) vce(robust) posttrend fig replace


       panel variable:  state_num (strongly balanced)
        time variable:  monthly, 2022m1 to 2022m12, but with gaps
                delta:  1 month

Iteration 1: tolerance = 1.449e-14

GEE population-averaged model                   Number of obs     =        561
Group variable:                  state_num      Number of groups  =         51
Link:                             identity      Obs per group:
Family:                           Gaussian                    min =         11
Correlation:                  exchangeable                    avg =       11.0
                                                              max =         11
                                                Wald chi2(3)      =      33.78
Scale parameter:                  22.92823      Prob > chi2       =     0.0000

                              (Std. Err. adjusted for clustering on state_num)
------------------------------------------------------------------------------
             |               Robust
_TRX_p~10000 |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          _t |  -.1256285   .0919356    -1.37   0.172     -.305819     .054562
    _x2022m7 |   1.981882   .4204735     4.71   0.000     1.157769    2.805995
  _x_t2022m7 |  -.0619396   .0949158    -0.65   0.514    -.2479712    .1240919
       _cons |   4.252497   .5037308     8.44   0.000     3.265203    5.239791
------------------------------------------------------------------------------


                    Postintervention Linear Trend: 2022m7

Treated: _b[_t]+_b[_x_t2022m7]
------------------------------------------------------------------------------
Linear Trend |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     Treated |  -.1875681   .0549749    -3.41   0.001     -.295317   -.0798192
------------------------------------------------------------------------------

Thanks,

[Jared Greathouse]

I'm sorry, I'm familiar with the econometrics and all the code here, but I don't get the issue. Yeah you can include year effects if you want! Nothing against it.

I don't get the second question though, could you please explain it a little more? Are you just asking if it's okay to reestimate the model at another point?

[Andrew Adkins Second]

I'm sorry, I'm familiar with the econometrics and all the code here, but I don't get the issue. Yeah you can include year effects if you want! Nothing against it.

I don't get the second question though, could you please explain it a little more? Are you just asking if it's okay to reestimate the model at another point?

I've been asked to calculate that immediate effect (m2022m7)--what happens immediately following the policy--to a specific month or the entire pre-period as opposed to the intercept (_cons). I'm just not sure if it's possible using some type of postestimation command such as lincom _b[_x2022m7]-something, or would I need to reorganize my data in such a manner that is amenable to that? I suppose the concern from them is that there are few time points and potentially some peaks there so they wanted to see how July 2022 compares to May 2022, for example, or how it changes relative to the average pre-period values.

[Jared Greathouse]

You can reestimate the model for May of 22 if you'd want, but I have additional questions that confuse me that I think are more important. Firstly, what units are treated versus untreated? As it currently stands, your syntax behaves as though you only have one unit and you're doing ITSA over that. But you dont'! You have panel data, or donors in a comparison group which you're comparing one unit to. So, of these 10 states, who is treated, and who is not treated? Once we can get past that, then we can discuss the other statsy details.

[Andrew Adkins Second]

You can reestimate the model for May of 22 if you'd want, but I have additional questions that confuse me that I think are more important. Firstly, what units are treated versus untreated? As it currently stands, your syntax behaves as though you only have one unit and you're doing ITSA over that. But you dont'! You have panel data, or donors in a comparison group which you're comparing one unit to. So, of these 10 states, who is treated, and who is not treated? Once we can get past that, then we can discuss the other statsy details.

The states make up a category of states that are impacted by the policy to varying degrees. Isn't single group ITSA, by design, have no comparable control group; rather, the preintervention trend projected into the treatment period serves as the counterfactual? So, then I use single group ITSA to estimate for all these states within this 1 group to estimate the impact of a large scale health intervention for example.

I'm now interested in seeing how the immediate impact in July compares to either May 2022 or the entire pre-period as opposed to the immediate impact relative to the intercept.

[Jared Greathouse]

Oh. I was under the impression that you had a control group. I really really think you should get one, if possible!!!!!! Anyways.

As far as I know, there's no direct way to test the impact from one month to another month. I don't even think ITSA is suited for this because it doesn't allow for effect heterogeneity. I think statistically and even philosophically, that might be challenging, and I don't have an answer here.

I wadogoing to suggest you just compare the intercept to the level change because, if I remember my ITSA math right, isn't that all the intercept is, the average pre-intervention trend as estimated? I could be totally wrong about this, but that's how I remember it being.....

[Andrew Adkins Second]

Oh. I was under the impression that you had a control group. I really really think you should get one, if possible!!!!!! Anyways.

As far as I know, there's no direct way to test the impact from one month to another month. I don't even think ITSA is suited for this because it doesn't allow for effect heterogeneity. I think statistically and even philosophically, that might be challenging, and I don't have an answer here.

I wadogoing to suggest you just compare the intercept to the level change because, if I remember my ITSA math right, isn't that all the intercept is, the average pre-intervention trend as estimated? I could be totally wrong about this, but that's how I remember it being.....

Yes, you're right, and thank you for your response. One additional conceptual question based on your experiences. For example, say you're examining the impact of a policy using ITSA like the cigsales ITSA example in the notes. What impact does removing the intervention month from the analysis and using the following month as the point of change--like a washout period.

[Jared Greathouse]

Is the idea that the treatment may not have "fully" taken into effect in the month, but actually "really" began in the following month?

[Andrew Adkins Second]

Is the idea that the treatment may not have "fully" taken into effect in the month, but actually "really" began in the following month?

Yes, essentially.