[Clyde Schechter]

The coefficient of dy/dx is 0.019 which shows the difference in differences of the outcome probabilities (outcome probabilities means probability of time and probability of mgnregadmy ). Please correct me if i am wrong.

You have the right idea. Your words are not quite accurate. In the probability metric, the average difference in differences in probability of loan20, is 0.019, or 1.9 percentage points if you prefer. This is, in probability terms, the difference in differences estimate of the effect of the treatment mgnregadmy.

Added: For the future, please learn to use code delimiters correctly. The instructions for using them in FAQ #12 seem clear enough and you should not encounter these difficulties if you follow them. Your PNG attachments do seem to come out well here, but that process is not entirely reliable. And sometimes those who want to help you would like to copy/paste part of your output to work with it, which a screenshot does not support.

[Neeraj Kumar]

Thanks for your suggestions and I really appreciate it. So this means my interpretation of interaction and time coefficients are right which I discussed in # 13. Pure intervention effect should be 0.119.

[Clyde Schechter]

Pure intervention effect should be 0.119.

I think you made a typo there. It's 0.019.

[Neeraj Kumar]

Thank you so much Dr. Clyde.

[Shenae Samuels]

Hello All:

I am new to the forum and I am seeking your assistance on this topic as well. I am using a difference-in-differences (DID) method with logit regressions. I am using data from the Behavioral Risk Factor Surveillance System (BRFSS) to examine the impact of a policy on binary outcome measures. The treatment variable is denoted using a factor variable as presented in the codes below:

Logit Regression Code

svy: logit insured i.post##i.exp other covariates

Code 2

margins post#exp

Code 3

marginsplot

Code 4

margins exp, dydx (post)

Code 5

margins exp, dydx(post) pwcompare (effects)

The issue that I am having is that the p-values between the DID coefficient in the logit model and the estimated DID in the predicted probabilities do not match. Is there perhaps a reason for this? Is there a different estimation method that I should be using?

Thank you all.

Shenae

[Clyde Schechter]

They will not, in general match. One might expect them to be in the same ballpark, but you can't expect anything more than that. Even that much will not necessarily be true. When they do come out very close to equal it's just a coincidence.

The coefficient in the logit model is estimating the logarithm of the ratio of odds ratios; it can also be thought of as the difference in differences of the log odds of the outcome. It is one way of looking at things. And to the extent that the logit model actually describes reality, it is a single statistic that summarizes the effect of the intervention.

The drawback is that the log odds metric is not only awkward and unfamiliar (except to people who do logistic regression frequently), but for decision-theoretic or policy-analytic purposes it isn't useful. For decision/policy purposes you have to know how many people would actually benefit from a policy. So the difference in differences of the outcome probability is the key measure of effectiveness in this context. But the same ratio of odds ratios can correspond to drastically different probability differences in differences depending on the baseline probabilities we are starting from.

Suppose that the probability of a desired outcome is .001 in the conrtol group, and .002 in the intervention group prior to the intervention. Suppose the OR (post:pre) in the control group is 1.0, but in the intervention group the OR is 1.5. That makes the ratio of odds ratios 1.5, which sounds impressive. But when you then calculate the post-intervention probabilities of a positive outcome it is 0.001 in the control group vs 0.003 in the intervention group. So the actual difference in differences in probabilities here is just two-tenths of one percentage point, which, unless the intervention is very cheap and has few or no side effects doesn't seem very worthwhile.

But keep the odds ratios the same and start with a pre-intervention period probability of 0.25 in the control group and 0.30 in the intervention group, and you find that post-intervention the probabilities are 0.3 and 0.391, respectively, representing a difference in differences of just over 9 percentage points, which is pretty considerable!

Remember both calculations are using the same within-group odds ratios and the same ratio of odds ratios.

So I think you can see from this simple example that a large (and statistically significant) odds ratio can correspond to a rather puny, and statistically insignificant difference in differences in the probability metric.

The situation in your output is actually even more complicated than that because the marginal effects (difference in differences in probability) are actually averaged over the entire sample, so just about anything can happen. But that just makes it even less reasonable to expect the p-values to be similar.

The good news is that a policy analyst, who will want to work in probability terms, generally won't even ask for a p-value. He or she probably will want confidence intervals around the estimates and might use them to do a sensitivity analysis on the decision, however.

[Shenae Samuels]

Thank you so much Dr. Schechter. Your explanation most certainly cleared up my concerns!

Best,
Shenae

[Neeraj Kumar]

I read the paper by Patrick A. Puhani "The Treatment Effect, The Cross Difference, and the Interaction Term in Non-Linear Difference in Difference Models". They mentioned that, " in a non-linear model such as probit, the cross difference (or derivative) does not represent the treatment effect and thus not an interesting parameter in a nonlinear "difference-in-difference" model. Instead, it is correct to focus on the coefficient of the interaction term". Like Mention in #18 the intervention effect (treatment effect) is measure by margin (dy/dx). So I am just little bit confuse with whom i have to go. coefficient of Interaction or margin (dy/dx) ?

[Clyde Schechter]

This is a matter of semantics, and, like many semantic issues, some people have strong opinions about it.

In a probit model, the coefficients, whether you have interaction terms or not, are really not interpretable. They only make sense to people who are comfortable with the cumulative normal distribution function, and are basically inexplicable to most of the world. In a logistic model the coefficients are logarithms of odds ratios, or, for interaction terms, logarithms of ratios of odds ratios. In some people's view, the ratio of the odds ratios is the "interesting" parameter for the DID model, so they focus on the coefficient of the interaction term. Those people are entitled to their view.

And the others are entitled to the opposite view. For us (I am now in this camp; in earlier years I was in the other camp), the problem with the ratio of odds ratios is that if the baseline outcome probability is close to 0 or 1, then even an enormous ratio of odds ratios (or an odds ratio) may correspond to a very tiny change in outcome probability. If one is working in the field of policy analysis, and if the purpose of the research is to inform decision making, it is the outcome probabilities that matter, and the odds ratio metric is a distortion. For that reason, it is the output of -margins-, and -margins, dydx()-, and -margins, dydx() pwcompare- that are the "interesting" parameters: they enable you to actually estimate the number of people, firms, counties (or whatever the unit of analysis in the study is) will actually experience an different outcome if the study policy is implemented.

You need to think about who your audience is, and present your findings in the way they will find most congenial. If you will have a mixed audience, it is probably best to present your findings both ways. And it is always important to be clear which statistics you are presenting when.

[Joseph Coveney]

Is is possible to get a statistically significant nonzero dydx() value at some set of covariate values of interest when the 1.treatment#1.post term in the probit or logit regression model is exactly zero?

[Neeraj Kumar]

Thanks Dr Clyde.

[Clyde Schechter]

Joseph Coveney Yes, you can. In fact, it is always possible to find a set of covariates such that the -margins, dydx()- at those covariates is zero. Of course, those covariates may be well outside the observed (or even possible) range of the actual variables. But sometimes they are within the realm of the data.

The question is, is this a bug or a feature? What is, from a mathematical perspective, the same dilemma comes up without even looking at sophisticated models. In health care research we are often interested in health disparities, where the incidence (or mortality) of some condition is greater in one population subgroup than in another. It is often the case that one tries some intervention and finds that the difference between the two group rates is not changed following the intervention, but the ratio (relative risk) has changed. Or the opposite can also happen. The question then is whether you consider the risk difference or the risk ratio the true measure of health disparity. Similarly here, you can have disparate results between looking at the interaction coefficient and looking at the difference in differences in the average probability. Which is the "correct" view? I think the answer is that it depends on how you plan to use and act on the results. As my career has meandered more into policy-analysis type work, it is clear to me that for decision making purposes the average probability metric is very important and the odds ratio is not. But that is just that perspective. As they say, where you stand depends on where you sit.

[Joseph Coveney]

So, what would the policy analysis recommendation be under those circumstances? The dydx() is positive and statistically significant for such-and-such a demographic segment, and logit's or probit's 1.treatment#1.post coefficient is unimpressive if not essentially zero?

I'm not trying to troll; I'm just not familiar with policy analysis, and how these kinds of circumstance—which apparently arise often enough—are handled. I assume that "for decision making purposes the average probability metric is very important and the odds ratio is not" would be tempered a little under such circumstances.

[Clyde Schechter]

Well, in a policy context we would generally not be dealing with one particular set of covariates. We would be looking at the average over the population (or perhaps one or more population subgroup) distribution of those covariates, i.e. the difference in average marginal effects. And I think that in a policy context, those would take priority over a zero or near zero coefficient in the log-odds metric, if the model is well calibrated to observed results. If the model is poorly calibrated, then I think a policy analyst would be reluctant to base a decision on any statistics derived from it.

[Clyde Schechter]

Let me give a more thoughtful reply to Joseph Coveney 's question in #28. Let's use a very simple situation. We have a single pre-intervention observation and a single post-intervention observation in both the treatment and control groups. We observe the proportion of positive outcomes in each of these four conditions. There are no other covariates.

Let's first note something about the (log) OR vs the probability DID. The concern raised is that one can be zero when the other is not. There is one circumstance that is necessary for this to be true. If the probability of positive outcome in the control group is the same both pre- and post-intervention, then, if the OR = 1 (logistic coefficient = 0), then the post-intervention positive outcome probability in the treatment group will also equal the post-intervention positive outcome probability in the treatment group, so that the probability DID will also be zero. In other words, in order for the probabliity DID and the log OR to disagree, there must be a difference in the positive outcome probability in the control group

Now let's look at a simple example. Suppose that in the control group the pre- and post- intervention positive outcome probabilities are .20 and .25, respectively. Then the pre-intervention OR is 1.333... Now suppose that in the treatment group the pre- and post-intervention positive outcome probabilities are 0.5 and 0.57. The post-intervention OR is easily seen to be (up to rounding error) 1.3333, so that in the logistic model, the interaction coefficient will be 0. But the outcome probability DID will be 0.02. If the sample size is sufficiently large, this difference will be statistically significant.

So the question becomes: do we believe there has been an intervention effect or not after seeing this data. If we rely on the logistic regression interaction coefficient, the answer is no. If we rely on the outcome probability DID the answer is yes. Note that if we had chosen to analyze this data with a linear probability model instead of a logistic regression, the interaction term coefficient would be 0.02 and there would be no controversy. So the same data leads to different conclusions depending on which model, or which model interpretation, you apply. This is the kind of subjectivity that arises all the time in frequentist statistical analysis but is usually swept under the rug. This particular situation forces us to examine it.

Now, what would a policy analyst say in response to this? The first thing he/she would probably notice is that there was a very large difference in the pre- and post- outcome probabilities in the control group; in fact this difference dwarfs the outcome probability DID. So what happened to the control group between pre- and post-intervention. Given that we have presumed a large enough sample size that the outcome probability DID of a mere 0.02 was statistically significant, we can pretty much exclude sampling error as the cause of this much larger difference. Perhaps the intervention "leaked" into the control group and they got "contaminated." Or perhaps we find out that there was a change in the way outcome was ascertained over time. In either case, then the study itself would be too flawed to support decision making on any analysis. If no design issues can be identified that invalidate the study, then we are left with the possibility that there is a secular trend in the outcome. (This is often actually the case in my field, epidemiology.) The secular trend, in this case, is quite large. But it suggests a way to refine the study to resolve the conflicting interpretations. We would want to go back and get multiple observations over time in both groups. If we had multiple pre-treatment observations in both groups, we could then observe how, before the intervention, the outcome probabilities in the treatment and control groups evolved. Recall that the classical assumption for DID analysis is parallel trends. But parallel trends in what? If we see that the treatment:control odds ratio remains more or less constant over time, then that would support reliance on the logistic regression coefficient. But if we see that the outcome probability difference between the two groups tends to remain more or less constant over time, then that would support relying on the outcome probability DID. That said, even if these additional data supported reliance on the outcome probability DID, when the demonstrated intervention effect is so small compared to secular trend, the policy maker might well decide that even though there is a significant effect, it is not large enough to warrant taking on the challenges and burdens of implementing the intervention at population scale. Statistical significance (whether of an OR or a probability DID) is never seen as an adequate criterion by itself; practical importance trumps that every time.

Finally, this is all a theoretical discussion. To construct such an example it is necessary to posit a large difference in the control group outcome probability in association with a constant OR (which, in turn, implies that the outcome probability DID is relatively small, though not 0). To get a situation where you find a relatively large outcome probability difference in the face of a constant OR you need to add covariates to the model, and then you need to go to extreme values of those covariates to create a substantial apparent paradox. These situations don't seem to arise much, if at all, in real world practice.