Interpreting marginal effects from multinomial logistic difference-in-differences (with interactions)

[Alice Cartwright]

I've found answers to many pieces of this question on this forum, but not one that quite gets at the challenge I'm having with correctly specifying marginal effects with a multinomial logit diff-in-diff that contains interaction terms.

In short, my outcome variable is a 4-level variable describing "type" of contraceptive method currently used. My data is repeated cross-sections over four waves of the National Survey of Family Growth, and I am trying to assess whether there were significant differences in the proportion of women using specific types of contraceptive methods in those states that expanded Medicaid in Jan 2014. I cannot post the exact output here, as I'm working with restricted data in a Federal RDC. However, here is a dummy example of the code and the output I'm getting and how I would interpret it. However, in an ideal world I would try to get the diff-in-diff value from the marginal effects (that would be interpretable the same way that an interaction coefficient between treatment status and being "post" the policy implementation from a linear regression/linear probability model would be).

Here's an example of the code used (in Stata 16) and example output:

svy, subpop(analysis12mo): mlogit method4cat treat##i.wave
margins, dydx(treat) at(wave=(1 2 3 4))

. margins, dydx(treat) at(wave=(1 2 3 4))

Conditional marginal effects                    Number of obs     =     21,609
Model VCE    : Linearized

dy/dx w.r.t. : 1.treat
1._predict   : Pr(method4cat==None_Barrier_methods), predict(pr outcome(0))
2._predict   : Pr(method4cat==Short_acting_methods), predict(pr outcome(1))
3._predict   : Pr(method4cat==LARC), predict(pr outcome(2))
4._predict   : Pr(method4cat==Permanent_methods), predict(pr outcome(3))

1._at        : wave            =           1

2._at        : wave            =           2

3._at        : wave            =           3

4._at        : wave            =           4

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
0.treat      |  (base outcome)
-------------+----------------------------------------------------------------
1.treat      |
_predict#_at |
        1 1  |  -.0159945   .0249968    -0.64   0.523    -.0652634    .0332745
        1 2  |  -.0057592   .0190036    -0.30   0.762    -.0432155     .031697
        1 3  |   .0011262   .0267605     0.04   0.966    -.0516189    .0538714
        1 4  |   .0379885   .0290085     1.31   0.192    -.0191875    .0951646
        2 1  |   .0078182   .0214355     0.36   0.716    -.0344313    .0500676
        2 2  |    .000771   .0182155     0.04   0.966    -.0351319     .036674
        2 3  |  -.0120649   .0209388    -0.58   0.565    -.0533354    .0292057
        2 4  |  -.0500352   .0165635    -3.02   0.003     -.082682   -.0173883
        3 1  |   -.007381   .0127026    -0.58   0.562    -.0324179    .0176559
        3 2  |  -.0147108   .0158917    -0.93   0.356    -.0460334    .0166119
        3 3  |  -.0182566   .0199098    -0.92   0.360    -.0574991    .0209858
        3 4  |  -.0066461   .0180239    -0.37   0.713    -.0421714    .0288791
        4 1  |   .0155573   .0222294     0.70   0.485    -.0282571    .0593717
        4 2  |   .0196989   .0194693     1.01   0.313    -.0186752    .0580731
        4 3  |   .0291953      .0237     1.23   0.219    -.0175177    .0759082
        4 4  |   .0186928   .0232084     0.81   0.421    -.0270511    .0644366
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

I would interpret this as: compared to states that did NOT experience Medicaid expansion (treat=0), the marginal effect of using "None/Barrier methods" at wave 1 is -0.0159945, or 1.6 percentage points lower. Then by wave 4, the marginal effect of using "None/Barrier methods" is 3.8 percentage points higher in treatment states. However, as I noted above, I don't think this output is giving me what I truly want which is the *difference* of pre and post levels in treatment states and pre and post levels in non-treatment states. I'd be grateful for any further thoughts/insight people can provide!

[Daniel Schaefer]

Alice, welcome to the forum, and thank you for the very readable formatting for the output.

what I truly want which is the *difference* of pre and post levels in treatment states and pre and post levels in non-treatment states.

I'm not sure what you mean here. The "treatment" variable indicates whether or not a state expanded Medicaid. It's not clear from your writeup if that happens in 2014 only, or if it's possible for a state to transition from no Medicaid expansion to having Medicaid expansion later. More importantly it also seems like, from your writeup, you are interested in pre-treatment levels and post-treatment levels of contraceptive use. So this quoted statement is a little unclear. What would it mean to have "post [treatment] levels" in non-treatment states?

Do you want to compare states with Medicaid expansion to those states without Medicaid expansion regardless of the wave?

I would interpret this as: compared to states that did NOT experience Medicaid expansion (treat=0), the marginal effect of using "None/Barrier methods" at wave 1 is -0.0159945, or 1.6 percentage points lower.

I might say that the log odds of using contraception type 1 in wave 1 in states that expanded Medicaid is 0.02 less than in the log odds of using contraception type 1 in wave 1 in states that did not expand Medicaid. I don't personally like the "percentage points lower" language for interpreting the value on the probability scale, since it is ambiguous mathematically, and can mean objectively different things to different people. It also implies subtraction, when the actual operation to calculate the difference (really a "factor change") is multiplication. e^(-0.0159945) ~= 0.984, meaning that the probability of using contraception type 1 in wave 1 in states that expanded Medicaid is 98.4 percent of the probability of using contraception type 1 in wave 1 in states that did not expand Medicaid.

[Alice Cartwright]

Thanks Daniel, this is really helpful, especially the feedback on interpretation of marginal effects in a non-linear model.

Some further clarification: for the ease of interpretation, I have excluded states that expanded Medicaid later in 2014/2015/2016 as they're a small number. So the comparison states are those that never expanded Medicaid through early 2019. I guess when I'm saying "post" levels in a non-treatment state, I'm referring to levels after 2014 in non-treatment states. My understanding of diff-in-diff is that this pre/post difference in non-treatment states is important to know, since there may have just been an increase in use of short-acting methods, for example, that was naturally occurring over time, regardless of any policy. Being able to determine that difference, as well as the difference pre/post in the treatment states, allows one to isolate the impact of the policy itself. The first wave of data was collected "pre" 2014, while the other 3 waves were after the policy implementation. So ideally I'd be comparing trends in states with Medicaid expansion to those states without Medicaid expansion, across all the post waves, with the one "pre" wave as the baseline or comparison.

Ultimately, I would want to know: if changes in the log odds in wave 2 (and 3 and 4) compared to wave 1 in the treatment states were significantly different from changes in the log odds in wave 2 (and 3 and 4) compared to wave 1 in the non-treatment states. I'm just not sure that's what I'm able to get from this output.

[Daniel Schaefer]

Well, it looks to me like we can say that most of these predicted probabilities of using a given contraceptive in states which adopted Medicaid expansion are not significantly different from the corresponding probability of using a given contraceptive in a state that did not adopt Medicaid expansion. But I agree, that's not quite what you are trying to get at. If I understand correctly, you want to be able to say that the treatment and control groups are moving away from wave 1 along different trajectories, with (perhaps) changing proportions of the use of each contraception year over year for the treatment group, but (perhaps) static proportions in the control group? And you need a hypothesis test to reject the null hypothesis that the trajectories are parallel?

I suspect that since most of the differences across treatment groups across waves are not significant in the first place, it is likely that the two groups (control and treatment) are not moving along statistically distinct trajectories in terms of the type of contraception used. But of course, my suspicions don't prove anything.

I'm afraid I'm not all that familiar with the DiD methodology, but based on this stats stackexchange thread it may be difficult to specify a DiD model for a logistic regression. Your mlogit model is a generalization of the logit, potentially complicating things further, and on top of that you have to account for the complex survey design and you have a subpopulation, meaning that you really have to account for the design or you risk very biased standard errors. I don't know with absolute certainty, but I suspect what you are asking may not be possible, or at least not possible in a way that is rigorously justified. Notably, Stata has a command for the linear DiD regression, but I can't find a generalized command for logit style regressions.

Hopefully, someone with more DiD experience will jump in.

[Daniel Schaefer]

You can see some ongoing discussion on this topic here: https://www.statalist.org/forums/for...sectional-data