Warning message for -permute- and -ritest- in Difference-in-Differences design

[Tarjei W. Havneraas]

Hi Statalisters

I'm doing a difference-in-differences (DiD) on a small sample (N=6, T=24) and I'm therefore interested in running permutation tests as several simulation studies recommend this (e.g. Rokicki et al. 2018).

However, I seem to run into some complications using Stata's -permute- and the user-written -ritest-. The documentation on the commands does not describe the warning messages(may be that it's clear but I'm just missing out as I'm new to permutation).

Specifically, when using the following DiD-model wit -ritest-, where intervention is binary treatment variable and post_17Q1 is binary post-variable,

ritest intervention _b[1.intervention#1.post_17Q1], reps(50) strata(region) ///
cluster(intervention): xtreg suicideRate i.intervention##i.post_17Q1, fe cluster(region)

I get:

Warning: some strata contain no variation in intervention

and,

Warning: 100% of the resampled realizations for _pm_1 are exactly identical to original value

When running -permute-,

permute intervention _b[1.intervention#1.post_17Q1], ///
reps(50): xtreg suicideRate i.intervention##i.post_17Q1, fe cluster(region)

I get no error message, but the results make no sense,

Monte Carlo permutation results                 Number of obs     =        144

      command:  xtreg suicideRate i.intervention##i.post_17Q1, fe cluster(region)
        _pm_1:  _b[1.intervention#1.post_17Q1]
  permute var:  intervention

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _pm_1 |  -.8296676       0       0       .       .         .          .
------------------------------------------------------------------------------
Note: Confidence interval is with respect to p=c/n.
Note: c = #{|T| >= |T(obs)|}
Note: Missing values observed in permutation replicates.

I've been working on this for some while and I can't seem to solve it. Hopefully some smart heads here on Statalist have some input.


References:
Rokicki et al. 2018. Inference with Difference-in-Differences With a Small Number of Groups: A Review, Simulation Study, and Empirical Application Using SHARE Data.

[Rich Goldstein]

the citation looks potentially interesting - what's the rest of the citation (e.g., journal name, volume number, pp.), please?

[Tarjei W. Havneraas]

Hi Rich

Yes, interesting read, and they also provide R code for their simulations. The full citation is,

Rokicki et al. 2018. Inference with Difference-in-Differences With a Small Number of Groups: A Review, Simulation Study, and Empirical Application Using SHARE Data. Med Care. 2018 Jan;56(1):97-105. doi: 10.1097/MLR.0000000000000830.

[Tarjei W. Havneraas]

As an update, I've worked some more on permutation and tried a new specification. My main aim is to assess whether the DiD interaction-term is significant. However, as I see from Clyde Schechter's comment in this post, my specification in #1 is probably incorrect.

With the following code I'm interested in whether the DiD-interaction term is significant,

permute suicideRate _b[1.intervention#1.post_17Q1], /// 
strata(region) reps(1000): xtreg suicideRate i.intervention##i.post_17Q1 i.quarter, fe 
(running xtreg on estimation sample)

Permutation replications (1000)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5 
..................................................    50
..................................................   100
..................................................   150
..................................................   200
..................................................   250
..................................................   300
..................................................   350
..................................................   400
..................................................   450
..................................................   500
..................................................   550
..................................................   600
..................................................   650
..................................................   700
..................................................   750
..................................................   800
..................................................   850
..................................................   900
..................................................   950
..................................................  1000

Monte Carlo permutation results

Number of strata =          6                   Number of obs     =        144

      command:  xtreg suicideRate i.intervention##i.post_17Q1 i.quarter, fe
        _pm_1:  _b[1.intervention#1.post_17Q1]
  permute var:  suicideRate

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _pm_1 |  -.8296676      46    1000  0.0460  0.0066  .0338715   .0608836
------------------------------------------------------------------------------
Note: Confidence interval is with respect to p=c/n.
Note: c = #{|T| >= |T(obs)|}

From what I've seen in the Stata help documentation, the p-value is 0.0460, which can be interpreted as evidence to reject the null hypothesis of no effect of the treatment. However, I'm not quite sure of this interpretation as I've not seen many examples of permutations in DiD-context (other than an R simulation in the above-mentioned reference, which is not easily transferable to Stata syntax in this case).