[Joseph Coveney]

So, it seems that the issue is moot, that in practice the interpretation based upon the model's interaction regression coefficient will be the same as that based upon the marginal effect in the probability metric, even when as usually the case there are covariates in the model. That sounds good to me, but it wasn't the impression that I had got earlier. Again, I am not familiar with this area, but I recall seeing (or at least I got the impression of) dire claims that the model's interaction term must not be used to assess effectiveness of an intervention, that you must use the treatment effect estimate in the probability metric, that they could even be opposite in sign.

[Thomas Stiles]

For that reason, it is the output of -margins-, and -margins, dydx()-, and -margins, dydx() pwcompare- that are the "interesting" parameters: they enable you to actually estimate the number of people, firms, counties (or whatever the unit of analysis in the study is) will actually experience an different outcome if the study policy is implemented.

I have a novice-type question about interpreting interactions in a logistic model, and it brought me to this thread. So far, I think the most helpful webpage has been the UCLA page: Deciphering Interactions.

So I made my novice attempt to interpret an interaction in an ed study, and am wondering whether this is the way to do it? The ratio of odds ratio was significant, but from what I gather from above and from elsewhere, graphing the simple effects, calculating the avg marginal effects, and then the interaction effect is maybe a better way of going about interpreting interactions? So here's my shot at doing this...

After the logistic command, I run margins...

.margins TeamLeader#Gender, post
.margins, coeflegend

I get the 'marginal probabilities' for the 'simple effects', which are plotted below.

So I write a statement: "Figure X depicts marginal probabilities for the simple effects of gender and team leader status on team member altering plan away from graduate school after team research." That's clunky, but is it an accurate statement?

Then I use lincom to calculate the 'average marginal effects' (or alternatively use the dxdy option)

.lincom (_b[1.TeamLeader#1.Gender ] -_b[1.TeamLeader#0bn.Gender])
.lincom (_b[0bn.TeamLeader#1.Gender] -_b[0bn.TeamLeader#0bn.Gender])

In this particular case, there were no significant 'average marginal effects'. So I write a simple statement: "The average marginal effects were insignificant."

Then following the protocol on the UCLA page, I calculate the 'difference in the average marginal effects' to get the 'interaction effect' on the probability for the outcome.

.lincom (_b[1.TeamLeader#1.Gender] -_b[1.TeamLeader#0bn.Gender]) - (_b[0bn.TeamLeader#1.Gender]-_b[0bn.TeamLeader#0bn.Gender])

Here, I do see a significant difference. So I write a statement: "However, the probability of a female team member with graduate team leader of altering her prior plan away from graduate school after team research is on average 0.24 less than other team members (SE = 0.11, z = −2.18, p < 0.05)."

So all together, the interpretive statements are: "Figure X depicts the marginal probabilities for the simple effects of gender and team leader status on team member altering plan away from graduate school after team research. The average marginal effects were insignificant. However, the probability of a female team member with graduate team leader of altering her prior plan away from graduate school after team research is on average 0.24 less than other team members (SE = 0.11, z = −2.18, p < 0.05)."

Does this approach to interpretation sound okay? Or am I butchering the statistical terminology?

Thank you for your help,
Thomas

[Clyde Schechter]

All of your proposed statements seem to be accurate interpretations of the graphs, and the are reasonably worded.

I have one suggestion. To some extent you are blending two different philosophies here. In my work (and not everybody agrees with me about this) I draw a sharp distinction between hypothesis testing and predictive modeling, and I have a strong preference for the latter over the former in most situations. The philosophies of those two endeavors are different, and the questions they think relevant to ask differ.

If you want to do hypothesis testing, I would argue that you shouldn't look at the marginal probabilities. Yes, you can do hypothesis tests about them, but, then again, you can also fry chocolate bars if you want to--but I wouldn't want to eat the results. If you want to test hypotheses, I would stay with the logistic regression results directly, and I would base my conclusions about the interaction on the p-value of the interaction coefficient, and I would base my conclusions about gender and team leader effects based on the p-values of the appropriate linear combinations of regression coefficients. The results are in the log-odds metric, but for these purposes that's just fine.

Now, in most situations, I don't like the hypothesis testing approach. I typically start from the premise that of course there is a gender effect, and of course there is a team leader effect, the relevant questions is their size and directions as manifested in their effects on the frequencies of outcomes. That would lead to the kind of analysis you have presented in your post here, with one tweak. I would not report the p-value because p-values have one and only one purpose in life: testing null hypotheses. If we agree that the null hypothesis here is probably a straw man, or, in any case, of no practical importance as a comparator, and we are interested in the size of differences in outcomes probabilities, then the p-values are out of place. So just the marginal effects and marginal effect differences with their standard errors would suffice. Or, you might throw in confidence intervals --just because most non-technical audiences will feel more comfortable with CIs than with standard errors (even though they are just different ways of expressing the uncertainty associated with the point estimates.)

[Montserrat Botey]

Hello ! This thread is very helpful for a paper I am writing using a control and treatment group after a Pension Reform. However, having myself gone through all the steps in #15, I was wondering if the following step is also crucial for determine if both groups (treatment and control groups) are statistically significant.
margins treatment, dydx(pre_post) pwcompare (group) For ex, looking at confidence intervals in 15 at the pwcompare step, I am confused if it makes the dydx significant given that they include 0.

Thanks a lot in advance for your help !

[Clyde Schechter]

Well, given that the American Statistical Association has recommended that the concept of statistical significance be abandoned (See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and
https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.), let's not discuss the question in these terms.

The output shown in #15 says that the estimated difference in differences is, to three decimal places, 0.019, with a 95% confidence interval from -.009 to +0.048. This tells us that the data do not support a precise enough estimate of the DID to make confident assertions about its sign, although its magnitude is clearly small, in whichever direction it lies. Now, depending on the actual substantive problem that led to that study and analysis, this may or may not be a useful result. If, for example, the nature of things wee such that only a difference of 0.1 or greater would be large enough to matter for practical purposes, we could, on this basis, say that the data are more compatible with the effect being too small to matter. On the other hand if an effect of 0.02 would be large enough to be of practical importance, then we can say that the data don't tell us enough, as they are consistent with an important effect, but also with an unimportant one. So the interpretation should rely on where an effect that is large enough to matter falls with respect to the confidence interval.

[Montserrat Botey]

Dear Clyde,

First of all, thanks a lot for your underlining that statistical significance is off the table now and the links.

Thanks a lot for your answer. At first I was very confused since I had output using the famous command for my differences in differences using a Logit model with interaction:

margins employe_prive , dydx(traite)

Where confidence intervals are fine :

And then....

margins employe_prive , dydx(traite pwcompare )

And then, while including the pwcompare option, we see that there is a not a " good" confidence interval. Conversely, I've had regressions where the first intra- groups margins command show intervals including the 0 and then an interval for the pairwise that does not include the 0. What should I look for? Pertinent intervals for both intra and inter group comparisons? Thanks and sorry if this is a question out of the scope of the forum (since it's also about statistics itself!!! And many many thanks again for your reply back in March !!

[Montserrat Botey]

Sorry for these horrible screenshots ( I just figured out how to post HLM).

margins employe_prive , dydx(traite)

1.traite
employe_prive
0	.09313	.0451346	2.06	0.039	.0046678	.1815923
1	.1295441	.0327278	3.96	0.000	.0653988	.1936894

margins employe_prive , dydx(traite) pwcompare

Contrast   Delta-method          Unadjusted
                            dy/dx         Std.Err      [95% Conf. Interval]

1.traite
employe_prive
1 vs 0	.0364141	.0495398	-.0606821	.1335102

[Nico Gatti]

You can do this:

logit i.treatment##i.pre_post other_covariates
margins treatment, dydx(pre_post) pwcompare

The contrast between the marginal effect of pre_post in the treatment and control groups is the average treatment effect in the probability metric: it is the difference in differences of the outcome probabilities. If a decision maker wants to evaluate a policy, and if the number or proportion of people (firms, entities, whatever) in the entire population that experience a positive outcome is a suitable utility metric, then it is this difference-in-differences that he or she would be interested in.

Of course, like any other effect in a non-linear model (it is the non-linearity of the model that is relevant here, not the discreteness of the outcome) the average marginal effect may not be a particularly useful statistic, depending on what your goals are. You may have greater need of marginal effects at particular levels of the covariates. But that can be accommodated as well by just adding an appropriate -at()- option to the -margins- command.

Dear all, I am using this code right now and I have two questions:

1. I would like to export the results in tex file, but I haven't managed to do so (I've added "post" after pwcompare and used esttab after saving the estimates).

2. I would like to include in my model also year fixed effects (i.year), but this makes it impossible to estimate the contrast between the marginal effect of pre_post in the treatment and control groups. Do you have any alternative suggestion on how to proceed?

Thanks a lot in advance!

Nicolò

[Clyde Schechter]

I can't help you with the export to a tex file--I don't do that myself and don't know how.

As for your second question, the problem arises because the pre_post variable is colinear with the year indicators. So in that context, the pre_post effect is unidentifiable: whatever result you get for it in the regression is an artifact of the choice of which year to omit in order to break the colinearity. So you can't use this model.

I take it your purpose is to estimate the causal effect of treatment in your data, and do so in the probability metric, but you feel that it is important to adjust for year-on-year shocks to the outcome. You can do that, but it requires a different model. You have to do it the same way you would do it if the onset of treatment were staggered in your data, with generalized DID.

gen byte active_treatment = treatment & pre_post
xtlogit outcome i.active_treatment i.year, fe
margins active_treatment, pwcompare

[Nico Gatti]

I can't help you with the export to a tex file--I don't do that myself and don't know how.

As for your second question, the problem arises because the pre_post variable is colinear with the year indicators. So in that context, the pre_post effect is unidentifiable: whatever result you get for it in the regression is an artifact of the choice of which year to omit in order to break the colinearity. So you can't use this model.

I take it your purpose is to estimate the causal effect of treatment in your data, and do so in the probability metric, but you feel that it is important to adjust for year-on-year shocks to the outcome. You can do that, but it requires a different model. You have to do it the same way you would do it if the onset of treatment were staggered in your data, with generalized DID.

gen byte active_treatment = treatment & pre_post
xtlogit outcome i.active_treatment i.year, fe
margins active_treatment, pwcompare

Thanks a lot! I am exploiting this model to estimate the causal effect of treatment, and I would also like to consider the heterogeneous effect among different age groups (I have 5 age groups, identified by a variable taking values 0 - 1 - 2 - 3 - 4).

My code is:

logit outcome i.active_treatment##i.age_group i.year additional_controls, r

margins active_treatment, at(age_group==0) pwcompare(effects)
margins active_treatment, at(age_group==1) pwcompare(effects)
margins active_treatment, at(age_group==2) pwcompare(effects)
margins active_treatment, at(age_group==3) pwcompare(effects)
margins active_treatment, at(age_group==4) pwcompare(effects)

Is it correct to interpret the obtained values as the causal effect for each different age group?

Thanks a lot!

[Clyde Schechter]

Yes, that would work. But there is a simpler way to get the results from -margins-:

margins age_group, dydx(active_treatment)

Also, in formally reporting this, you should refer to the values you obtain as the difference-in-differences estimates of the causal effect, not as the causal effect itself. Remember that the DID method of identifying causal effects relies on assumptions. The parallel trends assumption can be examined in the data. But there is also the assumption that the onset of treatment is not coincident with some other event that affects the outcome--and that is, by definition, something that cannot be verified from the data used in the analysis and, in general, can only be lent plausibility through arguments (hand-waving) about the context in which the study was carried out. So you should, in your descriptions, not claim to have identified the actual causal effect--you have used a method that is often successful at doing so, but the success rate in not 100% and you can't always distinguish the successes from the failures.

[Nico Gatti]

Yes, that would work. But there is a simpler way to get the results from -margins-:

margins age_group, dydx(active_treatment)

Also, in formally reporting this, you should refer to the values you obtain as the difference-in-differences estimates of the causal effect, not as the causal effect itself. Remember that the DID method of identifying causal effects relies on assumptions. The parallel trends assumption can be examined in the data. But there is also the assumption that the onset of treatment is not coincident with some other event that affects the outcome--and that is, by definition, something that cannot be verified from the data used in the analysis and, in general, can only be lent plausibility through arguments (hand-waving) about the context in which the study was carried out. So you should, in your descriptions, not claim to have identified the actual causal effect--you have used a method that is often successful at doing so, but the success rate in not 100% and you can't always distinguish the successes from the failures.

Thank you so much!