Treatment effect significant for a subpopulation?

[Julian Hackinger]

We analyse the effect of a policy change in soccer on the incidence of injuries. Due to our theoretical predictions, we hypothesise that the treatment effect depends on the asymmetry of a match. Therefore, in a difference-in-differences design, we use a logit model to estimate the conditional probability of an injury y on the interaction of the variable indicating our treatment group (treated), the variable indicating the periods following the introduction of the policy (post), and the asymmetry of the match, a continuous variable (asymmetry), hence (treated##post##c.asymmetry).

We get a significant overall treatment effect (treated#post), however a just not significant coefficient for the interaction of the treatment effect with the asymmetry of the match (treated#post#asymmetry). Nevertheless, plotting the margins, one can see that the treatment leads to an increase in injuries in asymmetric matches (see figure). Given the figure, we are actually quite surprised that treated#post#asymmetry is insignificant.

Does anyone have an idea what kind of additional tests we could run and report to demonstrate and capture this treatment effect on the subpopulation of sufficiently asymmetric matches?

[Felix Bittmann]

I guess your problem here are the large standard errors, which lead to the insignificant result. An easy solution would be to repeat the analyses using only the cases with top 30% of asymmetry. Otherwise you could also recode this metric variable into terciles and quartiles and work with that.

[Julian Hackinger]

Thank you Felix Bittmann! We decided to report the results of the margins output table of the treatment effect which demonstrates that the treatment effect is significant for very asymmetric matches.